# Homelab Documentation > GitOps-driven Kubernetes homelab on bare metal, running Flatcar Container Linux, Kubeadm, Cilium, Rook-Ceph and ArgoCD. Bare-metal Kubernetes on Flatcar Container Linux. Ansible and a PXE boot server bring the nodes up; Argo CD owns everything after that. The pages are reference material: architecture decisions, one page per platform component, operations runbooks and the contribution workflow. # Get started # Flatcar Homelab ![Homelab Logo]() A rack of machines and a pile of YAML: a bare metal Kubernetes cluster that provisions itself over PXE, keeps its entire state in Git, and is documented here so that nothing has to be re-derived at 02:00. It is a real cluster doing real work, built the way a production cluster is built, at a scale where breaking it is a learning experience rather than an incident review. ## Start here - **I want to build this** --- [Adapt It to Your Cluster]() first — the repository URL, domain and addresses are hardcoded — then the [Quickstart](). Set aside an afternoon. - **I know Kubernetes, not this repo** --- [Architecture]() for how it fits together, [Design Decisions]() for why, and [Known Limitations]() for what it does not do. - **I am learning Kubernetes** --- [Core Concepts]() covers the five ideas underneath this cluster — immutable OS, first-boot provisioning, sysexts, network boot, GitOps — and what each one costs. - **Something is broken** --- [Operations]() has the health check and the symptom-to-cause table. If it is a stalled install, the [PXE troubleshooting table](). - **I am making a change** --- [Development]() for the checks a pull request must pass, [GitOps Strategy]() for what a merge does to the cluster, and the [repository layout]() for where things live. ## The stack in one table Each choice had a simpler alternative that was rejected on purpose; the reasoning and what each one costs is in [Design Decisions](). | Layer | Choice | Why it is not the obvious one | | --- | --- | --- | | OS | [Flatcar Container Linux]() | Read-only `/usr`. You cannot `apt install` your way out of a problem, which turns out to be the feature | | Cluster | Kubernetes via kubeadm | Stock upstream, so the upstream docs apply verbatim | | Network | [Cilium]() | eBPF, no `kube-proxy`, no iptables archaeology. Also supplies Gateway API and the LoadBalancer addresses bare metal does not come with | | Storage | [Rook-Ceph]() | Replicated block storage across the nodes, and the component most likely to teach you humility | | Secrets | [OpenBao]() | Nothing sensitive in Git, at the price of a manual unseal after every reboot | | Delivery | [ArgoCD]() | If it is not in Git it is not real, and it will not survive the next reconcile | The sections follow the reader's journey: [Get Started]() to build it, [Architecture]() to understand it, [Platform]() for each of the components, [Operations]() to run it, and [Development]() to work on the repository. # Core Concepts Five ideas carry this cluster, and none of them is Kubernetes; Kubernetes is the ordinary part. What makes the project unusual is how the machines underneath it come to exist and stay current. Read this before the [Quickstart]() if any of the names in the table are new. Skip it if none of them are. | Concept | What it is | What it costs you | | --- | --- | --- | | [Flatcar Container Linux](<#flatcar-container-linux>) | An immutable OS with a read-only `/usr` and A/B partition updates | No `apt install`. Anything unusual has to arrive another way | | [Ignition & Butane](<#ignition-and-butane>) | First-boot provisioning from a JSON config | It runs *once*. Changing a template means rebuilding the node | | [Systemd sysexts](<#systemd-sysexts>) | Read-only images that extend `/usr` at boot | Upgrading Kubernetes means swapping an image and rebooting | | [PXE boot](<#pxe-boot>) | Booting a machine off the network instead of its disk | Needs a DHCP server you control and a host on the same segment | | [GitOps](<#gitops>) | Git is the desired state; a controller reconciles to it | If it is not committed, it does not exist | ## Flatcar Container Linux [Flatcar]() is an immutable, minimal Linux distribution designed for running containers. The root filesystem is read-only — you cannot install packages or modify system files at runtime — which forces all configuration to happen declaratively, at first boot, through Ignition. A read-only `/usr` makes undocumented drift between "identical" nodes impossible rather than merely discouraged. Updates are downloaded in the background onto the passive half of an A/B partition pair, so a bad one can be rolled back and nothing changes until the machine restarts. **Nothing here reboots itself**: this project masks `locksmithd`, the daemon that would normally coordinate that, and hands the job to [Kured](), which drains the node first. See [Updates & Upgrades](). ## Ignition and Butane **Ignition** is Flatcar's first-boot provisioning system. It reads a JSON config and applies it: creates users, writes files, partitions disks, enables systemd units. **Butane** is the human-writable YAML that compiles to that JSON. Ansible renders Butane from Jinja2 templates on the deployment host and transpiles it; the boot server serves the result. > [!WARNING] > > **Ignition runs once, in the initramfs, before the real root is mounted** > > It is not a configuration management system and it converges nothing on the second boot. Change a template and the node has to be *reprovisioned* to care — which here means arming it with `make reinstall` and network-booting it, wiping the disk on the way in. The config is embedded into the OEM partition at install time, so an installed node reads it from its own disk rather than from the network. ## Systemd sysexts Because `/usr` is read-only, software the base image does not ship — only `kubernetes`, in this cluster — arrives as a **system extension**: a read-only squashfs image overlaid onto `/usr` at boot, managed by `systemd-sysupdate`. Nodes fetch it from the HTTP boot server on their first boot from disk. containerd is part of the base image and is deliberately not replaced: a sysext-bakery build would run a containerd Flatcar has not tested. Upgrading Kubernetes on these nodes is therefore not `apt upgrade`; it is swapping an image and rebooting. See [Updates & Upgrades](). ## PXE boot **PXE** (Preboot Execution Environment) lets a machine boot from the network instead of a local disk. The NIC asks DHCP where to go, fetches a bootloader over TFTP, and the bootloader fetches everything else. In this project: 1. An external DHCP server hands out the boot server's IP and a syslinux filename. 2. TFTP serves the syslinux bootloader and a per-MAC boot menu. 3. Syslinux fetches the Flatcar kernel and initrd over HTTP, passing the Ignition config URL as a kernel parameter. TFTP runs over UDP with no error correction worth the name, which is why everything larger than the bootloader moves to HTTP as fast as possible. The full sequence is in [Boot & Bootstrap Process](). ## GitOps Everything the cluster runs is described in this repository under `payload/`, and [ArgoCD]() continuously reconciles the cluster to match. If it is not in Git, it is not in the cluster — and if you put it in the cluster anyway, the next sync of the Application that owns it puts Git's version back. Two terms recur throughout this site: - **ApplicationSet** — one ArgoCD object that generates an `Application` per matching file in the repository, so adding a component is adding a directory. - **Sync policy** — every `Application` syncs automatically, reverts drift and retries a failed sync, so nothing orders the platform: a fresh cluster converges as CRDs, storage and secrets appear. The order it settles in is in [Platform → Rollout order](); the policy itself is in [GitOps Strategy](). # Quickstart ![Homelab Logo]() Bring up the bare metal Kubernetes cluster from scratch, using Flatcar Container Linux and Kubeadm. The steps are thirteen commands; set aside an afternoon anyway, because step 7 depends on your DHCP server behaving. > [!WARNING] > > **Read this first if the cluster isn't mine** > > Node names, IP addresses, the `wlkr.ch` domain, and the Git repository URL are hardcoded throughout `payload/`. Following the steps below verbatim gives you a cluster whose ArgoCD syncs from **this** repository and whose certificates are issued for a domain you don't control. Work through [Adapting This for Your Cluster]() **before** step 1. ## Hardware Requirements Each node must have: - A NIC that supports PXE booting - An NVMe drive, or a different `install_disk` in `inventory.yaml` - Enough disk for Flatcar and the root filesystem. On workers the root partition is capped (`size_mib` in `ansible/templates/butane_node_config.yaml.j2`) and Rook-Ceph takes the remainder as OSD storage; control planes keep the whole disk - At least 4 GB of RAM: the installer runs from a RAM disk and streams the Flatcar image straight to disk The deployment host (the machine running Ansible and the boot server) must be on the same L2 network segment as the nodes. ## Prerequisites ### Tools on the deployment host | Tool | Used by | Install | | --- | --- | --- | | `make` | every step | Xcode CLT / `build-essential` | | `git` | cloning this repository | your package manager | | `uv` | Ansible and the boot server | `curl -LsSf https://astral.sh/uv/install.sh \\| sh` | | `butane` | transpiling Butane YAML to Ignition JSON | `brew install butane` or the [Flatcar docs]() | | `gpg` and `gpgv` | checking the signatures on the Flatcar and syslinux checksums in `make download` | `brew install gnupg` or your package manager | | `kubectl` | steps 8 onwards | [kubernetes.io]() | | `helm` | `make install-cilium`, `make install-argo` | [helm.sh]() | ### Other requirements - **SSH Key**: An Ed25519 key at `~/.ssh/id_ed25519.pub` (or edit `ansible/templates/butane_node_config.yaml.j2` and `butane_installer_config.yaml.j2` to use a different path/key) - **External DHCP Server**: Must point PXE clients at the deployment host and give every node a fixed address: - Option 66 (`next-server`): IP of the machine running `make serve` - Option 67 (`filename`): `lpxelinux.0` for BIOS, `syslinux.efi` for UEFI - A reservation per `mac_address` handing out exactly that host's `ansible_host`. Nothing in the node config sets a static address; the API server's `advertiseAddress`, the etcd peer URLs kubeadm derives from it and the metrics listeners are all rendered from `ansible_host`, so a node that leases a different address joins with the wrong one or not at all. > [!TIP] > > **Consumer routers and PXE** > > Many home routers accept options 66 and 67 and then serve neither. If step 7 produces silence in the boot server log, prove the DHCP side first with `tcpdump -i port 67 or port 68`. ## Setup 1. **Clone Repository**: ```bash git clone https://github.com/JanWelker/homelab.git homelab cd homelab ``` 2. **Configure Inventory**: Edit `ansible/inventory.yaml` to define your target nodes and settings. Every variable, and the `make` target that bakes it into `output/`, so you know what to re-run after an edit: | Variable | Why it matters | Read by | | --- | --- | --- | | `boot_server_ip` | **Most commonly missed.** Baked into the generated PXE menu as the URL for the kernel, initrd, and Ignition config. If this isn't the IP of the machine that will run `make serve`, nodes load the bootloader and then hang. | `make config`, `make serve` | | `mac_address` (per host) | Selects which generated PXE menu a node picks up | `make config`, `make reinstall`, `make serve` | | `ansible_host` (per host) | The address the node's DHCP reservation hands out; rendered into the kubeadm and static pod configs | `make config`, `make reinstall` | | `install_disk` | Target disk for the Flatcar install, overridable per host for a node whose disk sits elsewhere. **Wiped completely** — partitions, GPT and a device-level discard — before the install | `make config` | | `control_plane_vip`, `control_plane_vip_interface` | The API endpoint kubeadm bakes into every certificate and kubeconfig, and the NIC kube-vip advertises it on; see [Control Plane VIP]() | `make config`; the VIP also `make kubeconfig`, `make reinstall` | | `control_plane_endpoint_host` | The name or address nodes join through; a name only on a fresh build, see [Endpoint name]() | `make config`, `make kubeconfig`, `make reinstall` | | `pod_subnet`, `service_subnet` | kubeadm's cluster networks; change only on a collision with your LAN | `make config` | | `router_ip`, `router_asn`, `cluster_asn` | The eBGP peer the Gateway addresses are announced to; rendered into `output/router/frr-bgp.conf` for the router, and `payload/platform/cilium/bgp.yaml` must say the same; see [Cilium]() | `make config` | | `flatcar_channel`, `flatcar_version`, `syslinux_version` | The OS image and bootloader to fetch; `flatcar_version` also names the image the installer writes | `make download`; `flatcar_version` also `make config` | | `kubernetes_version` | The sysext image to fetch, and the version kubeadm and sysupdate are rendered with | `make download`, `make config` | | `kube_vip_version` | The bootstrap static pod only; the running image is pinned in `payload/platform/kube-vip/`; see [Inventory]() | `make config` | | `bootstrap_cluster` (`control_plane` group) | `true` for this first install, so the first control-plane node runs `kubeadm init`; back to `false` once the cluster is up — see [Inventory]() | `make config`, `make reinstall` | | `control_plane` / `workers` membership | Decides the node role, and with it the partition layout and the kubeadm command | `make config` | | `ansible_user`, `ansible_ssh_common_args` (`k8s_nodes` group) | How the playbooks reach a running node for the kubeconfig and the join-credential renewal | `make kubeconfig`, `make config` | 3. **Initialize Environment**: Creates the `uv` virtual environment and installs dependencies: ```bash make setup ``` 4. **Download Artifacts**: Fetches Flatcar, Syslinux and the Kubernetes sysext image into `output/http`: ```bash make download ``` 5. **Generate Configurations**: ```bash make config ``` Writes Ignition configs to `output/http` and PXE menus to `output/tftp`. Each host gets `ignition--install.json`, which the PXE environment runs to wipe the disk and install, and `ignition-.json`, which `flatcar-install` embeds into the installed system — see [Boot & Bootstrap Process](). Re-run this after **any** change to `inventory.yaml`; the values are baked into the generated files. `make artifacts` runs steps 4 and 5 together. Check that one PXE menu was generated per node: ```bash ls output/tftp/pxelinux.cfg/ # one 01- file per host ``` 6. **Start Boot Server**: ```bash make serve ``` Leave this running through step 7 and keep the window visible: it names each node and logs one line per request. Its opening lines say what each node will do when powered on: a boxed `ARMED TO INSTALL` warning names the nodes that will wipe and install, and a node under *booting from disk* will not, however often you reboot it. ```console ⚠ ARMED TO INSTALL: odin, thor Every disk above is wiped on boot -- the partition table, every partition on it, and the Ceph OSD with them. No data survives, and nothing asks for confirmation at the console. To cancel, run: make reinstall-cancel make reinstall-cancel LIMIT= for one 20:33:04 server booting from disk: freya, heimdall, loki, valkyrie 20:34:17 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting its boot menu -- armed, so it will install 20:34:48 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting the OS image (1.2 GB) -- this is the long one 20:36:12 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c OS image delivered -- switching to local boot, so the reboot lands on the disk ``` 7. **Arm the install, then boot the machines**: ```bash make reinstall ``` The generated PXE configs default to booting the local disk; `make reinstall` flips them to `Install`, for `LIMIT=` or, after a typed confirmation, for every host. The boot server disarms a node itself once it has delivered the OS image, so the reboot at the end of the install lands on the disk — see [Switching back to local boot](). Then power on the nodes. No menu appears; each node does what it was armed to do, and the boot server log walks it through the fetches, then `switching to local boot`, then after the reboot the Ignition config again plus the sysext images. A node still shown by bare IP has not fetched a menu the server recognises. - **Leave the boot server running until every node is up.** The sysexts are fetched on the first boot from disk. - The cluster comes up `NotReady` because no CNI is installed. This is correct; do not fix it yet. - If a node stalls, see [Troubleshooting PXE boot](<#troubleshooting-pxe-boot>). > **Watching an install** > > `journalctl -u flatcar-install -f` as `core@` (`flatcar-install-uefi` on UEFI firmware) shows the wipe and the write (so does a monitor on the node). A failed install does *not* reboot — the node stays in the PXE environment with the journal intact. 8. **Retrieve Kubeconfig**: Once the control plane node responds to SSH: ```bash make kubeconfig ``` Every `make` target that touches the cluster reads `output/kubeconfig` (`make KUBECONFIG= ` picks another); only `kubectl` in your own shell needs the export. Verify the cluster answers. `NotReady` is expected at this point: ```bash export KUBECONFIG="$PWD/output/kubeconfig" kubectl get nodes ``` *Optional: install to the local machine (will not overwrite an existing config):* ```bash mkdir -p ~/.kube cp -n output/kubeconfig ~/.kube/config ``` 9. **Install Cilium** (CRITICAL): ```bash make install-cilium ``` Installs the Gateway API CRDs, then Cilium via Helm — only what ArgoCD needs in order to run; everything else arrives through ArgoCD, at the versions pinned in each component's `application.yaml` (see [GitOps Strategy → Version pins]() and [Cilium → Installation]()). `make bootstrap` runs this step, `make install-argo` and `make bootstrap-apps` in one go. Nodes reach `Ready` once Cilium is up: ```bash kubectl -n kube-system rollout status ds/cilium kubectl get nodes # all Ready ``` 10. **Post-Installation**: - **Deploy ArgoCD**: ```bash make install-argo ``` There is no way into the UI yet: the local admin is disabled in `payload/argocd/values.yaml` and Authentik arrives later through GitOps. Follow the bootstrap with `kubectl` instead. **Tip — the UI before Authentik exists:** re-enable the local admin with a patch. It holds only until the next step: right now ArgoCD is the plain Helm release `make install-argo` made, and `make bootstrap-apps` hands it to the self-healing `argocd` Application, whose first sync puts `admin.enabled` back to what `payload/argocd/values.yaml` says. After that the only way in is the pull request in [Authentik → When Authentik is down](). The restart is what generates the password Secret: ```bash kubectl -n argocd patch cm argocd-cm --type merge \ -p '{"data":{"admin.enabled":"true"}}' kubectl -n argocd rollout restart deploy/argocd-server kubectl -n argocd rollout status deploy/argocd-server kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath='{.data.password}' | base64 -d; echo kubectl -n argocd port-forward svc/argocd-server 8080:80 # http://localhost:8080, user admin ``` Nothing to undo: the handover reverts the patch. - **Hand the cluster over to ArgoCD**: ```bash make bootstrap-apps ``` Applies the AppProjects, then the self-managing `argocd` Application, which brings the `platform` ApplicationSet and with it every component at once. They converge in the [order their dependencies allow](); watch them settle: ```bash kubectl -n argocd get applications -w ``` **Everything that reads a secret stays Degraded, and that is expected.** No `ExternalSecret` resolves until step 11 — see [Bootstrap pauses at OpenBao](). - **Gate on storage** before trusting anything that mounts a volume: ```bash make storage-check ``` A `StorageClass` exists whether or not a CSI driver registered for it, so `rook-ceph-cluster` can go Healthy with no working provisioner. This asks for a volume the way a workload would and names the first broken link — see [Rook-Ceph → Is storage ready?](). 11. **Initialise the secret store**: OpenBao starts uninitialised, sealed and empty; every `ExternalSecret` stays Degraded until it is unsealed and `certificates` until it is populated, then everything converges on its own. ```bash make bao-init ``` Initialises and unseals OpenBao, enables the `kv` v2 engine and the Kubernetes auth method, and writes the policy and role External Secrets authenticates with, then replaces the root token with an operator login. The keys land in `output/credentials/openbao-init.json`, the login in `openbao-operator.json` beside it. > **Move those keys before you do anything else** > > That file is a plaintext copy of the keys to every secret the cluster holds. Copy them into a password manager and delete it. Losing them means the data is unrecoverable. See [OpenBao → Bootstrap](). Then populate the paths the cluster reads: ```bash make bao-secrets ``` It prompts, with input hidden, for every value that belongs to an account outside the cluster: the DNS, mail and other third-party logins listed at the top of `scripts/bao-secrets.sh`, each against a path in [OpenBao → KV layout](). A required value has no skip; an optional one is issued by something that runs on the cluster and takes Enter until it exists. A fork without one of those accounts removes its prompt from the script. Everything random — `kv/authentik/config` with the OIDC client credentials ArgoCD and Grafana read back, Grafana's break-glass admin password — is not prompted for: a `PushSecret` writes it once the store validates, see [Generated secrets](). On a running cluster it asks per path before overwriting — see [Rotating a credential](). > **Paste them at the prompt, not onto a command line** > > A secret containing `#` on a command line is truncated at it, and one containing `!` is mangled by history expansion — both silently. Non-interactively, the matching environment variables are honoured when already set; quote them with **single** quotes. Once the store validates, every `ExternalSecret` resolves, `certificates` issues the gateway certificates and the Gateways come up. The store is re-checked every few minutes, so that can take that long to start. 12. **Create the first administrator**: Authentik ships the built-in `akadmin` account, and the blueprints switch it off as soon as it exists — see [Bootstrap credentials](). Switch it on, and read its password back out of OpenBao (needs a token — `bao login -method=userpass username=operator` inside the pod, or the port-forward in [OpenBao → Authenticating locally]()): ```bash kubectl -n authentik exec deploy/authentik-server -c server -- ak shell -c \ 'from authentik.core.models import User; User.objects.filter(username="akadmin").update(is_active=True)' kubectl -n openbao exec openbao-0 -- \ bao kv get -mount=kv -field=bootstrap-password authentik/config ``` Log in at [auth.k8s.wlkr.ch]() as `akadmin`; the first login enrols a second factor. Create a personal account in *Directory → Users* and put it in `authentik Admins` and the groups listed in [Authentik → Groups and roles]() under *Directory → Groups*; the blueprints create them. ArgoCD's `policy.default` is empty, so an SSO login that lands on an ArgoCD with no applications means a missing group, not a broken integration. Then switch `akadmin` off again with the same command and `is_active=False`; when Authentik itself is unavailable, see [When Authentik is down](). ## Single-node clusters The [documented layout]() has dedicated workers, so the control-plane taint stays. On a single node: - Remove the taint once `make kubeconfig` has answered and **before `make install-argo`**. Cilium tolerates the taint, but ArgoCD does not, so `make install-argo` would wait on unschedulable pods until Helm's `--wait` times out: ```bash make untaint ``` - Set `redis-ha.enabled: false` in `payload/argocd/values.yaml`; its hard per-host anti-affinity leaves pods Pending on one node whatever the taints say. - Re-apply the taint when you later add worker nodes: ```bash make taint ``` ## Reprovisioned nodes The installer wipes the disk before `flatcar-install` runs, so a node armed with `make reinstall` hands Ceph an empty disk. If that wipe ever fails, the symptom is no `rook-ceph-osd` pods, a `CephCluster` still reporting `Ready`, and `openbao-0` Pending on an unbound PVC at step 11 — see [Rook-Ceph → No OSDs after reprovisioning](). ## Verifying the result The five commands that answer "is it actually fine?": ```bash kubectl get nodes # all Ready kubectl -n argocd get applications # all Synced / Healthy kubectl get certificate -A # READY=True kubectl -n rook-ceph get cephcluster # HEALTH_OK make storage-check # a PVC actually binds ``` The last one earns its place: `cephcluster` reports `Ready` on a cluster with no OSDs and no CSI driver. Once DNS points at the gateway IPs, the platform UIs are reachable — see [Platform → HTTPRoute Locations](). ## Troubleshooting PXE boot Step 7 fails as a black screen with a blinking cursor and no error message. Read the `make serve` log instead of the node and ask how far it got. | Symptom | Likely cause | | --- | --- | | Nothing after the startup lines; the node never appears | DHCP isn't handing out options 66/67, or the node isn't on the same L2 segment. Check the DHCP lease and that PXE is enabled in firmware. | | Node comes up on an address other than its `ansible_host` | No per-MAC DHCP reservation, or one for the wrong NIC. The join uses the rendered address, not the leased one. | | `PXE-E32: TFTP open timeout` | `make serve` isn't running, or a firewall is blocking UDP/69. On macOS, allow the Python interpreter through the firewall. | | Bootloader loads, then "Could not find kernel image" or a hang at the menu | `boot_server_ip` in `inventory.yaml` is wrong. It is baked into the menu's kernel/initrd URLs. Fix it, re-run `make config`, and reboot the node. | | `no generated menu has that MAC` in the boot server log | The node's `mac_address` in `inventory.yaml` doesn't match its actual NIC. The log prints the MAC the node actually asked for; put that in the inventory and re-run `make config`. | | `collecting its boot menu -- booting from its local disk` | The node is not armed. `make reinstall LIMIT=` and boot it again. Expected after an install: the boot server disarms a node once it has the OS image. | | Kernel boots, then Ignition fails | The node couldn't fetch `ignition--install.json` over HTTP (port 8000), or the Butane template references an SSH key path that doesn't exist. | | `refusing to wipe active disk` on the first boot after an install | The installer's disk stanza reached the installed system. `wipe_table` belongs in `ignition--install.json` only — see [Wiping the disk](). | | Node installs but never joins the cluster | Sysext download failed, or the join unit errored. SSH in as `core` and check `journalctl -u bootstrap-k8s`. | A machine with two NICs PXE boots from whichever it likes, and the MAC on the case is often not that one. A node that boots the menu and stalls right after is almost always a wrong `boot_server_ip`, not a broken image. The boot server serves `output/tftp` over TFTP and `output/http` over HTTP — see [Boot & Bootstrap Process](). If a file is missing from those directories, re-run `make artifacts`. # Adapting This for Your Cluster This repository documents one specific homelab. Node names, addresses, the `wlkr.ch` domain, and the Git repository URL are hardcoded throughout `payload/` and `ansible/`. Run the [Quickstart]() unchanged and you get a cluster that syncs from *this* repository and requests certificates for a domain you don't control. Everything below is a change you make in your own fork, before step 1 of the Quickstart. ## 1\. Fork and repoint ArgoCD Every ArgoCD `Application` points at this repository by URL. Until you change them, your cluster pulls its desired state from here and your own commits have no effect. ```bash git grep -l 'github.com/JanWelker/homelab.git' -- payload/ ``` Rewrite them to your fork. The match ends in `.git`, so the workloads repository below is left alone: ```bash git grep -lz 'github.com/JanWelker/homelab.git' -- payload/ | \ xargs -0 sed -i '' 's|github.com/JanWelker/homelab.git|github.com/YOUR_USER/YOUR_REPO.git|g' ``` The workloads repository is a second URL, in `payload/platform/argocd-projects/projects.yaml` (the `apps` project's `sourceRepos`) and `payload/workloads/applicationset.yaml`. Point both at your fork of [homelab-apps](). > [!NOTE] > > `sed -i ''` is the BSD/macOS form. On Linux use `sed -i` with no argument. If your fork is private, ArgoCD also needs repository credentials — see the [ArgoCD private repository docs](). ## 2\. Choose your domain Four hostname patterns carry traffic. Each is a listener in `payload/platform/gateway-api/gateways.yaml` and a zone in external-dns's `domainFilters` (`payload/platform/external-dns/application.yaml`); the HTTPS ones also have a wildcard `Certificate` in `payload/platform/certificates/certificates.yaml`: | Pattern | Purpose | In a fork | | --- | --- | --- | | `*.k8s.` | user-facing workloads, via `apps-gateway` | rename | | `*.infra.k8s.` | platform UIs (ArgoCD, Grafana, Hubble, OpenBao, Rook), via `infra-gateway` | rename | | `*.ssh.` | the [SSH listener]() for in-cluster agent sessions | rename, or delete the listener and the zone | | `*.wollbi.ch` | a second site family on `apps-gateway`, with its own listener and certificate | delete or rename every file `git grep -l wollbi -- payload/` lists | Replace the first three in one pass; `wollbi.ch` does not match the pattern and is handled by hand: ```bash git grep -lz 'wlkr\.ch' -- payload/ | \ xargs -0 sed -i '' 's|wlkr\.ch|YOUR-DOMAIN.example|g' ``` That covers the Gateway hostnames, the wildcard `Certificate` resources, every `HTTPRoute`, ArgoCD's `global.domain`, and the `link.argocd.argoproj.io/external-link` annotations. You must own this domain: the certificates are issued by Let's Encrypt through a DNS-01 challenge, which requires write access to the zone. `.local`, `.lan` and made-up TLDs will not work. ## 3\. Point DNS at the gateway IPs The two Gateways take fixed addresses from the Cilium pools in `payload/platform/cilium/lb-pools.yaml`: | Gateway | DNS record | | --- | --- | | `apps-gateway` | `*.k8s.` | | `infra-gateway` | `*.infra.k8s.` | Change both CIDRs to free addresses on your LAN, outside the DHCP range, or the router will eventually hand one to a laptop, then create the two wildcard `A` records. Two more things in the same component decide whether anything answers on those addresses: - The `CiliumL2AnnouncementPolicy` in `lb-pools.yaml` answers ARP for them from the control-plane nodes only, on interfaces matching a name pattern. Change the `nodeSelector` if your control planes should not carry the traffic and `interfaces` if the pattern does not match your NICs; L2 needs the addresses in the nodes' subnet. - `payload/platform/cilium/bgp.yaml` also announces them to the router over eBGP, and its peer and ASNs must match the inventory (step 5). With no BGP router, delete the file. ## 4\. Set up the DNS-01 solver `payload/platform/certificates/cluster-issuers.yaml` is written for **AWS Route53**. Update: - `email:` — your address, on both issuers. Let's Encrypt sends expiry notices here. - `region:` — the Route53 region. If your DNS is hosted elsewhere, replace the `dns01.route53` solver with the matching [cert-manager DNS-01 provider]() and adjust the credential path in OpenBao accordingly. > [!TIP] > > Switch `issuerRef` in `certificates.yaml` to `letsencrypt-staging` while you are still iterating. Production allows 5 duplicate certificates per week, and a misconfigured solver burns through that in minutes. ## 5\. Describe your hardware Edit `ansible/inventory.yaml`; every variable is listed under [Quickstart → step 2](). The ones a fork must choose rather than copy: | Setting | Notes | | --- | --- | | `control_plane_vip` | Free address on the nodes' subnet, outside DHCP and distinct from the LoadBalancer pools. Becomes the API endpoint in the cluster certificates. | | `control_plane_vip_interface` | The NIC kube-vip advertises on; check `ip link` on a provisioned node. | | `control_plane_endpoint_host` | Set a name on a fresh build, see [Endpoint name](); a running cluster keeps the VIP. | | host entries | Replace the hosts with yours. Each needs `ansible_host` and `mac_address` (the NIC that PXE boots), and a DHCP reservation that pairs the two: the node takes its address from DHCP, and the kubeadm configs are rendered from `ansible_host` — see [Quickstart → Other requirements](). | | `router_ip`, `router_asn`, `cluster_asn` | The eBGP peer for the Gateway addresses; `payload/platform/cilium/bgp.yaml` must name the same values, since the payload cannot read the inventory — see [Cilium](). | Then update `payload/platform/cilium/values.yaml`: - `k8sServiceHost` — a literal, reachable API server address, because Cilium replaces `kube-proxy` and cannot reach the API through a Service — see [Design Decisions → Cilium](). - `devices` — the interface prefix Cilium binds to; check `ip link` on a provisioned node. And `payload/platform/kube-vip/daemonset.yaml`, which takes over from the bootstrap static pod once ArgoCD syncs and cannot read the inventory: - `address` — the same value as `control_plane_vip`. - `vip_interface` — the same value as `control_plane_vip_interface`. A mismatch here does not fail at provisioning. It fails on the first sync, when the DaemonSet removes the static pods and starts advertising somewhere else. ## 6\. Provisioning access Both Butane templates, `ansible/templates/butane_node_config.yaml.j2` and `butane_installer_config.yaml.j2`, inject `~/.ssh/id_ed25519.pub` as the authorized key for the `core` user. Point both at your own key if you use a different path or algorithm. There is no password, no console login, and no rescue path short of reprovisioning, so this is the only way into the nodes afterwards. ## 7\. The documentation site If you want your fork to publish its own copy of these docs, update `zensical.toml`: `site_url`, `repo_url`, `repo_name`, and `copyright`, and `docs/CNAME`, which binds the Pages site to a custom domain; delete it if you have none. The `docs.yaml` workflow then publishes to your own GitHub Pages. Otherwise, delete `.github/workflows/docs.yaml` to stop the build from running. ## Checklist Before `make config`: - \[ \] `repoURL` points at your fork in all of `payload/`, and the workloads URLs at yours - \[ \] Domain replaced throughout `payload/`, `wollbi.ch` renamed or removed - \[ \] LoadBalancer IPs are free addresses on your subnet, and the L2 policy or BGP peer announces them - \[ \] Wildcard DNS records created for both gateways - \[ \] ACME email and DNS-01 provider match your setup - \[ \] `inventory.yaml` describes your nodes, with the right `boot_server_ip` - \[ \] `control_plane_vip` is free, and `control_plane_vip_interface` matches the NIC - \[ \] `address` and `vip_interface` in the kube-vip DaemonSet match those two - \[ \] `k8sServiceHost` and `devices` match your control plane and NICs. On a new build, point `k8sServiceHost` at `control_plane_vip` once the VIP answers — see [Control Plane VIP]() - \[ \] SSH public key path is correct - \[ \] Changes committed and pushed — ArgoCD reads from Git, not your working tree # Architecture # Architecture A bare metal Kubernetes cluster on Flatcar Container Linux and kubeadm: a rack of machines, a network, and a deployment host that talks them into existence. The stack itself is summarised in [one table]() on the home page, and the reasoning behind each choice in [Design Decisions](). This section assumes you know what Flatcar, Ignition, systemd sysexts, PXE and GitOps are; [Core Concepts]() covers all five. ## What is in this section - **[Boot & Bootstrap Process]()** --- One node from power-on to `kubectl get nodes`, arrow by arrow, and the boot server that serves each step. Start here when an install stalls. - **[Ansible]()** --- The playbooks that generate everything the boot server serves. Ansible never configures a node. - **[GitOps Strategy]()** --- The ApplicationSet, the sync policy every Application shares, and how a fresh bootstrap converges without an order. - **[Design Decisions]()** --- Each choice as *X, not Y*, with its cost. Read before proposing a simpler alternative. - **[Security Posture]()** · **[Audit Logging]()** --- The trust boundary this cluster assumes, what is deliberately not enforced, and what the API server records. - **[Known Limitations]()** --- What it does not do, in one table. ## Cluster Layout The deployment host — the laptop running Ansible and the boot server — is **not** part of the cluster and nothing depends on it once provisioning is done. | Role | Nodes | Purpose | | --- | --- | --- | | Control plane | odin, thor, loki | etcd and the API server; tainted, no workloads, no Ceph OSD | | Worker | freya, heimdall, valkyrie, baldur, tyr, frigg, bragi, idun | Everything else, and a Ceph OSD partition each | Three control-plane nodes because etcd needs an odd number, and two is the worst answer: twice the hardware and quorum still lost when one dies. The API server is reached through a kube-vip virtual IP rather than any single node — see [Control Plane VIP](). Addresses, the VIP and the pod and service subnets are in `ansible/inventory.yaml`. ## Repository layout `ansible/` describes the machines, `boot_server/` hands them their operating system, `payload/` is the platform the cluster runs on, and `output/` is generated — never edit anything in there, the next `make config` overwrites it. The applications the cluster exists to serve live in the separate [`homelab-apps`]() repository; see [GitOps Strategy](). ```text . ├── .github/workflows # CI: linters, docs build and preview, argo-diff-preview -- see Maintenance ├── ansible # Inventory, playbooks and templates -- see Ansible ├── boot_server │ └── serve.py # TFTP + HTTP boot server ├── docs # This site ├── output # Generated; never edited by hand │ ├── credentials/ # Bootstrap token, certificate key, etcd │ │ # encryption key, OpenBao unseal keys -- 0700 │ ├── http/ # Ignition, Flatcar artifacts, sysext images │ ├── kubeconfig # Admin kubeconfig │ ├── router/ # FRR BGP config for the router │ ├── tftp/ # PXE bootloader and menus │ └── tmp/ # Readable copies of the rendered Butane and kubeadm configs ├── overrides # Theme template overrides for the docs site ├── payload # Everything ArgoCD deploys │ ├── argocd/ # ArgoCD itself: Application, ApplicationSet, values │ ├── platform/ # One directory per component, each with one │ │ └── ... # application.yaml -- see Platform │ └── workloads/ # The handover to homelab-apps ├── scripts # What the make targets run: OpenBao bootstrap, storage check, fonts ├── tests # pytest for the boot server ├── Makefile # Every operator command, one target each ├── renovate.json # The version pins Renovate tracks -- see Maintenance ├── zensical.toml # Documentation site configuration └── README.md ``` `ansible/` and `boot_server/` matter only while a node is being built; `payload/` matters every day after that. The exception is `ansible/templates/kubeadm.yaml.j2`, which explains why half the control plane is configured the way it is. `output/credentials/` is generated once and read back on every later run; `make config` writes new values for anything missing, which a running cluster will not accept. `make clean` therefore prints that inventory and asks before deleting, and refuses without a terminal. | Target | Removes | Keeps | | --- | --- | --- | | `make clean-artifacts` | `output/http`, `output/tftp`, `output/tmp`, `output/router` | `output/credentials/`, `output/kubeconfig` | | `make clean` | everything under `output/` | nothing — it asks first | # Ansible Configuration Ansible never configures a node here: it generates files on the deployment host, and Ignition applies them once, on the first boot after a node is installed. It renders two Ignition configs per host from two templates — one the PXE environment [runs to install](), one `flatcar-install` embeds into the system it writes. Changing anything under `ansible/` therefore takes a rebuild of the affected node, not a reboot. The directory layout is in [Repository layout](). ## Inventory `ansible/inventory.yaml` is the single source of truth for the machines; every generated artifact comes from it, and the values are baked into the output rather than read at boot. The variables are described in [Quickstart step 2](). Every playbook targets the `k8s_nodes` group, the parent of `control_plane` and `workers`, which carries `ansible_user` and the SSH options. > [!WARNING] > > **Every edit needs a regeneration, and some need two** > > `make config` re-renders the Ignition configs, PXE menus and router config from the inventory; `make download` fetches what `flatcar_channel`, `flatcar_version`, `syslinux_version` and `kubernetes_version` name. The Flatcar, Kubernetes and kube-vip versions are rendered into the configs too, so a version bump needs `make artifacts`, which runs both. Which target reads which variable is in the [Quickstart table](). An edit followed by neither changes nothing at all, silently. `bootstrap_cluster` in the `control_plane` group decides whether the first control-plane host renders `kubeadm init` (`true`, the first install) or, like every other control plane, `kubeadm join --control-plane` (`false`, every rebuild). A rebuilt node that ran `init` would start a second cluster on the same VIP — see [Rebuilding a node](). `kube_vip_version` only sets the bootstrap static pod that gives `kubeadm init` a VIP; once ArgoCD syncs, `payload/platform/kube-vip/` takes over and its image is what runs — see [Control Plane VIP](). ## Playbooks | Playbook | Does | | --- | --- | | `config.yaml` | Generates the bootstrap token, certificate key and etcd encryption key into `output/credentials/` (mode `0700`); renders the Ignition configs via Butane into `output/http` (mode `0600`), the sysupdate config for running nodes beside them, the per-MAC PXE menus into `output/tftp/pxelinux.cfg/`, the router's FRR config into `output/router/frr-bgp.conf`, and a readable copy of the kubeadm config into `output/tmp/` (the copy that runs is inlined into Ignition) | | `download.yaml` | Fetches the Flatcar kernel, initrd, OS image and its detached signature, checked against the `.DIGESTS.asc` that `gpgv` verifies; the Kubernetes sysext image; and the syslinux bootloader files, checked the same way against `sha256sums.asc`. The signing keys are fetched and trusted through the primary fingerprints pinned in the playbook | | `reinstall.yaml` | Rewrites `DEFAULT` in the generated PXE menus — `install` to arm, `localboot` to disarm. `make reinstall` and `make reinstall-cancel` are the two directions, both taking `LIMIT=`. Arming refuses while `bootstrap_cluster` is `true` and the API VIP answers, or when an [endpoint name]() is not the one the cluster publishes. It touches only `output/tftp/pxelinux.cfg/`, so the next `make config` regenerates the safe default. See [Switching back to local boot]() | | `kubeconfig.yaml` | Copies the admin kubeconfig over SSH to `output/kubeconfig`, mode `0600`, with the VIP as its `server` | `config.yaml` is safe to re-run: the `password` lookup reads an existing credential file back instead of generating a new value, so the encryption key every control-plane node shares stays the one etcd's Secrets were encrypted with. `config.yaml` renews the join credentials on the first control-plane host that answers with a running cluster, so the node being [replaced]() may be any of them; it fails when none answers while `output/kubeconfig` exists, since a node booted then could not join, and skips the renewal silently only before the first bootstrap. `kubeconfig.yaml` targets `control_plane[0]` — the first control-plane host in the inventory, with no fallback. If that node is down, copy `/etc/kubernetes/admin.conf` off any other control-plane node by hand. Keep a copy of the result off the cluster: it is the credential you want when the web UIs are unreachable, and `output/` is exactly as durable as the laptop it is on. # Audit Logging The API server records who did what, to which object, and whether it was allowed. It is also the destination for the `audit` half of the Pod Security Admission labels in [`pod-security.yaml`](): without it, findings at the stricter audit level go nowhere. ## At a glance | | | | --- | --- | | Policy file | `/etc/kubernetes/audit/policy.yaml`, written by Ignition from `ansible/templates/butane_node_config.yaml.j2` | | Log file | `/var/log/kubernetes/audit/audit.log` on each control-plane node | | Durable copy | [Loki](), via the Alloy pipeline in `payload/platform/alloy/application.yaml`, on Ceph | | Query it with | `{job="kubernetes-audit"}` in Grafana | | Worst-case disk | Bounded by the rotation flags in `ansible/templates/kubeadm.yaml.j2`, see [Where the log lives](<#where-the-log-lives>) | ## The policy decides everything `--audit-log-path` on its own does nothing: without `--audit-policy-file` the API server declines to open a log, silently. Rules are evaluated top to bottom and **the first match wins**, so the order is the design: | Matched | Level | Why | | --- | --- | --- | | `/healthz*`, `/livez*`, `/readyz*`, `/version`, `/metrics`, `/openapi*` | `None` | Polled continuously by kubelets, probes and Prometheus; the majority of requests | | Leases, Events | `None` | Renewed every few seconds by every component | | Reads by the control plane and by `system:nodes` | `None` | Not the reads anyone looks for; dropping them makes the next row affordable | | Secrets, ConfigMaps, TokenReviews | `Metadata` | Reads included: a *read* is the interesting verb | | `pods/exec`, `pods/attach`, `pods/portforward` | `RequestResponse` | Which pod, as which user, running what | | Everything else read-only | `None` | | | Everything that changes state | `Metadata` | Also the level that carries the Pod Security Admission annotations | > [!CAUTION] > > **Never raise the Secret rule above `Metadata`** > > At `Request` or `RequestResponse` the request body is recorded, and for a Secret the body *is* the credential. The audit log would become an unencrypted copy of every Secret in the cluster, in a flat file beside the etcd that was [encrypted at rest]() to prevent exactly that. ## Where the log lives | Property | Value | | --- | --- | | Path | `/var/log/kubernetes/audit/audit.log` | | Filesystem | root, ext4 | | Rotation | The `--audit-log-maxsize`, `--audit-log-maxbackup` and `--audit-log-maxage` flags in `ansible/templates/kubeadm.yaml.j2`; the product of size and backups bounds the footprint, and the age cap keeps a quiet cluster from holding a month-old rotation forever | The root filesystem persists across reboots, so the log survives on the node. Alloy still tails it into Loki, which is the copy that matters: a log stored only on the node is unavailable in precisely the situation where the node is what failed. ## Reading it Audit events reach Loki as JSON with `job="kubernetes-audit"`, plus `verb` and `audit_level` as labels, set by the Alloy pipeline in `payload/platform/alloy/application.yaml`. Everything else stays in the line, where `| json` can reach it: ```text # Who read Secrets, and which ones {job="kubernetes-audit"} | json | objectRef_resource="secrets" # Every exec into a running container {job="kubernetes-audit", verb="create"} | json | objectRef_subresource="exec" # Pod Security Admission violations that were audited but not enforced {job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations" ``` The last query answers "what would break if I tightened `enforce` on this namespace" from evidence, for the whole retention window. Exec, Secret reads by a person, RBAC writes, 403 bursts and anonymous requests also fire alerts from Loki's ruler, defined in `payload/platform/logging/loki-rules.yaml`; see [Logging](). ## Applying it to a running cluster Changing the templates changes what a **newly provisioned** node gets. A running control-plane node keeps the `kube-apiserver.yaml` static pod manifest kubeadm rendered at init time, and nothing rewrites it on its own. > [!CAUTION] > > **Get the policy file there before you re-render the manifest** > > An API server started with `--audit-policy-file` pointing at a missing file does not start. Step 3 before step 4, one node at a time, confirming the API server comes back before moving on. 1. Regenerate the Ignition configs so a rebuild gets this too: ```bash make config ``` 2. Create the directories on the node: ```bash ssh core@ sudo mkdir -p /etc/kubernetes/audit /var/log/kubernetes/audit ssh core@ sudo chmod 0700 /etc/kubernetes/audit /var/log/kubernetes/audit ``` 3. Place the policy file. It is inline in the Butane template, so take it from the rendered copy: ```bash yq '.storage.files[] | select(.path == "/etc/kubernetes/audit/policy.yaml") | .contents.inline' \ output/tmp/butane/butane-.yaml > /tmp/policy.yaml scp /tmp/policy.yaml core@:/tmp/policy.yaml ssh core@ sudo install -m 0600 /tmp/policy.yaml /etc/kubernetes/audit/policy.yaml ``` 4. Replace the install-time kubeadm config on the node with the regenerated one, then re-render the API server static pod from it: ```bash scp output/tmp/kubeadm/kubeadm-config-.yaml core@:/tmp/kubeadm-config.yaml ssh core@ sudo install -m 0600 /tmp/kubeadm-config.yaml /opt/kubeadm-config.yaml ssh core@ sudo kubeadm init phase control-plane apiserver \ --config /opt/kubeadm-config.yaml ``` 5. Watch the kubelet restart the static pod: ```bash kubectl -n kube-system get pod kube-apiserver- -w ``` 6. Update the `kubeadm-config` ConfigMap in `kube-system` so the flags survive the next `kubeadm upgrade` or control-plane join — kubeadm reads the ConfigMap on those paths, not `/opt/kubeadm-config.yaml`: ```bash kubectl -n kube-system edit configmap kubeadm-config ``` # Boot & Bootstrap Process One node from the power button to `kubectl get nodes`. When an install stalls, the question is *which arrow did not happen?* ## 1\. Preparation (on the deployment host) `make config` generates the per-host Ignition and PXE configs; `make serve` starts the [boot server](<#boot-server>). The [Quickstart]() has the exact sequence. ## 2\. Network Boot ``` sequenceDiagram participant Node participant DHCP participant Server as Boot Server Node->>DHCP: 1. PXE request DHCP-->>Node: 2. IP, next-server, filename Node->>Server: 3. TFTP bootloader Server-->>Node: 4. syslinux + menu Node->>Server: 5. HTTP kernel, initrd Server-->>Node: 6. Flatcar kernel, initrd ``` The DHCP server is external and must hand out the boot server's IP as `next-server` and a syslinux filename — see the [Quickstart prerequisites](). Step 2 is where most first attempts die, silently: the firmware moves on to the next boot device. ## 3\. Install & Bootstrap PXE boots the **installer**, and only on a node armed with `make reinstall`; the generated menu otherwise says `DEFAULT localboot`. The installer is a RAM environment whose only job is to write Flatcar to `install_disk` and reboot into it. Two templates, because two machines share a disk and nothing else: | Template | Renders to | Runs in | `wipe_table` | | --- | --- | --- | --- | | `butane_installer_config.yaml.j2` | `ignition--install.json` | the PXE environment | `true` | | `butane_node_config.yaml.j2` | `ignition-.json` | the installed system, first boot | `false` | The PXE menu points at the first. The installer fetches the second as a plain file and hands it to `flatcar-install -i`, which embeds it in the OEM partition of the system it writes. > [!TIP] > > **Keep the installer minimal** > > Anything in `butane_installer_config.yaml.j2` is downloaded into RAM on every install of every node and discarded ninety seconds later. If it configures the node rather than the installation, it belongs in `butane_node_config.yaml.j2`. ``` sequenceDiagram participant Node participant Server as Boot Server Node->>Server: 7. HTTP Ignition config Note over Node,Server: Only when armed with make reinstall Server-->>Node: 8. ignition--install.json Node->>Server: 9. HTTP Flatcar image + signature Note over Server: Server disarms the menu back to localboot Node->>Node: 10. Wipe disk, flatcar-install, reboot Note over Node: Now booting from disk, not the network Node->>Node: 11. Ignition partitions and writes /etc Node->>Server: 12. HTTP sysext image Server-->>Node: 13. kubernetes Node->>Node: 14. systemd unit runs kubeadm Note over Node: Node is NotReady - no CNI yet ``` Between steps 9 and 10 the boot server rewrites the node's menu to `DEFAULT localboot` — see [Switching back to local boot](<#switching-back-to-local-boot>). From step 11 the node runs from disk and needs the boot server only for the sysexts in step 12, on this first boot. `NotReady` after step 14 is expected: there is no CNI until `make install-cilium`. ### Wiping the disk Step 10 is the point of no return, and the wipe has three parts: | Wipe | Where | What it does | | --- | --- | --- | | `wipe_table: true` | `ignition--install.json` only | Destroys the GPT before the installer runs, so partition numbers and offsets come out the same whether the disk was empty or held the last cluster. Without it a rebuild inherits the old layout and `ignition-disks.service` fails on an `sgdisk` overlap | | `blkdiscard` | installer | Returns the device to unwritten. Best-effort: SATA without TRIM declines | | `wipe_filesystem: true` on `rook-osd` | `butane_node_config.yaml.j2` | Erases the old BlueStore signature, which `ceph-volume` reads instead of the partition table; a repartition alone can resurrect an OSD from a previous cluster | The installed system's config must not wipe: Ignition refuses to touch the disk it booted from (`refusing to wipe active disk`) and the first boot fails before anything is written. `storage.filesystems` follows the same split — `rook-osd` does not exist in the installer, and Ignition would block waiting for it. Check `install_disk` before you check anything else. ### Boot order The install leaves the firmware's boot order alone (`flatcar-install -u`, which would put the disk at the front of `BootOrder`, is not passed). The nodes network-boot first and reach their disk through `LOCALBOOT`, so the generated menu decides every boot; a disk promoted ahead of PXE would make `make reinstall` rewrite a menu the firmware no longer reads. A node therefore needs network boot first with a working `LOCALBOOT`, or the disk ahead of PXE in the firmware — with neither it installs and then has nothing to boot. ## 4\. Post-Installation Bootstrap Once kubeadm has initialised the control plane, `make bootstrap` installs only what ArgoCD needs to run, and ArgoCD itself. Once the `argocd` Application is applied, the repository is in charge. ``` sequenceDiagram participant Admin as Operator participant Deploy as Deployment Host participant Cluster Admin->>Deploy: 1. make install-cilium Deploy->>Cluster: 2. Helm install Gateway API CRDs, Cilium Note over Cluster: Nodes become Ready Admin->>Deploy: 3. make install-argo Deploy->>Cluster: 4. Helm install ArgoCD Admin->>Deploy: 5. make bootstrap-apps Deploy->>Cluster: 6. Apply AppProjects + the argocd Application ``` > [!NOTE] > > **`make untaint` is not part of this flow** > > It removes the control-plane `NoSchedule` taint and applies only to a single-node cluster, where it goes *before* `make install-argo` — ArgoCD has no tolerations. The [documented layout]() has dedicated workers, so the taint stays. See [Single-node clusters](). ## Every boot after the first The node boots from its own disk; the boot server should be off. | | | | --- | --- | | `/` | ext4 on partition 9, grown into it by `grow-root.service`: a fixed size on workers (`size_mib` in `ansible/templates/butane_node_config.yaml.j2`) so the OSD partition gets the rest, the whole disk on control planes | | Ignition | Runs **once**, on the first boot after the install | | `/etc/kubernetes`, `/var/lib/etcd`, `/var/lib/rook` | On disk; survive a reboot | | `rook-osd` | Partition 10 on workers, raw and unmounted — Ceph owns it | `bootstrap-k8s.service` does not fire again: its `ConditionPathExists=!/etc/kubernetes/kubelet.conf` is no longer satisfied. > [!NOTE] > > **Why `grow-root.service` exists** > > Taking partition 9 over in Ignition stops Flatcar's stock `systemd-growfs-root.service`: it is pulled in by an `x-systemd.growfs` mount option that a root mounted from `root=LABEL=ROOT` does not carry. Without the unit the node keeps the image's \~1.6 GB filesystem inside a partition many times that size. It runs the same binary and is a no-op once the filesystem fills the partition. > [!NOTE] > > **Two boot paths, and the menu picks the safe one** > > `PROMPT 0` boots whatever `DEFAULT` names, and the template always emits `DEFAULT localboot`, so a node that network-boots for any reason lands on its own disk. `make reinstall` rewrites that one line to arm it; see [Rebuilding or repartitioning a node](). Holding Shift or Alt at boot forces the prompt, the escape hatch for a node armed by mistake. ## Boot server `boot_server/serve.py` is a TFTP server and an HTTP server, installed by `uv sync` as the `boot-server` command. `make serve` runs it as you, never root, from the repository root (the document roots hang off `--root`, which defaults to the working directory): macOS denies Local Network access to a process that started as root and dropped it, so every TFTP reply would fail with *No route to host*. HTTP binds `boot_server_ip` from `ansible/inventory.yaml` — the address `make config` baked into every generated URL — and nothing else, because it serves the Ignition secrets. TFTP listens on every interface, the only way an ordinary user may bind port 69 on macOS; on Linux, lower `net.ipv4.ip_unprivileged_port_start` to 69. If no interface holds `boot_server_ip` it says so and names the variable; if port 8000 is taken it exits. A failed send ends that node's transfer, not the server. `--bind`, `--http-port` and `--tftp-port` override the address and ports, which is how the tests and a dry run on a laptop avoid the privileged port. | Server | Root | Serves | | --- | --- | --- | | TFTP, port 69 | `output/tftp` | `lpxelinux.0` (BIOS) or `syslinux.efi` (UEFI), and the menus in `pxelinux.cfg/` | | HTTP, port 8000 | `output/http` | Kernel, initrd, OS image and signature, Ignition configs, sysext images. No directory listings | Both roots are written by `make config` and `make download`; a missing file is fixed by `make artifacts`, not by the script. Leave it in the foreground. It opens by naming what it serves and which nodes are armed — a boxed warning listing every disk about to be wiped, with `make reinstall-cancel` and its `LIMIT=` form — then logs one line per request against the node that made it: role, name, address and MAC once the menu it fetched has identified it, dashes before. The file a node *stops* at is the diagnosis; see [Troubleshooting PXE boot](). ```console 20:33:04 server http on 10.9.200.222:8000 from output/http 20:33:04 server tftp on every interface, port 69, from output/tftp ⚠ ARMED TO INSTALL: odin, thor Every disk above is wiped on boot -- the partition table, every partition on it, and the Ceph OSD with them. No data survives, and nothing asks for confirmation at the console. To cancel, run: make reinstall-cancel make reinstall-cancel LIMIT= for one 20:33:04 server booting from disk: freya, heimdall, loki, valkyrie 20:34:17 - - 10.9.2.1 - collecting the bootloader (lpxelinux.0) 20:34:17 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting its boot menu -- armed, so it will install 20:34:19 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting the kernel 20:34:21 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting the initrd (391.2 MB) 20:34:48 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting its Ignition config 20:34:48 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting the OS image (1.2 GB) -- this is the long one 20:36:12 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c OS image delivered -- switching to local boot, so the reboot lands on the disk 20:38:40 control-plane odin 10.9.2.1 fc:3f:db:0d:8f:5c collecting the kubernetes sysext (61.4 MB) ``` A node is identified by MAC from the `01-` menu it fetches, and after the install by name from the `ignition-.json` it asks for. A request from an address that has done neither is logged against the bare IP. ## Switching back to local boot `make reinstall` writes `DEFAULT install` into a node's menu and nothing in the generated files writes it back, so firmware that network-boots first would reinstall forever. The boot server therefore disarms the node the moment it has delivered the whole OS image — the same edit `make reinstall-cancel` makes. A node that never fetched an Ignition config cannot be identified by name; the server says so and tells you to run `make reinstall-cancel` yourself. Arming survives the boot server, because the menu is a file on disk. `Ctrl-C` therefore checks: if anything is still armed it offers `Disarm 2 node(s) now? [Y/n]`. Enter accepts; anything else leaves them armed and repeats the cancel command. Without a terminal on stdin it warns and leaves them armed. > [!WARNING] > > **It disarms on delivery, not on success** > > The server sees a transfer complete, not whether `flatcar-install` then wrote the disk. An install that fails after the download leaves a disarmed node with no working disk — loud, because `flatcar-install.service` (`flatcar-install-uefi.service` on UEFI firmware) does not reboot on failure — and the fix is `make reinstall LIMIT=` before the power cycle. > [!WARNING] > > **Stop it when you are done** > > While it runs, anything on the segment can fetch the Ignition configs, which embed the kubeadm bootstrap token and certificate key — together enough to join a control-plane node. The token expires in 24 hours (`ttl` in `ansible/templates/kubeadm.yaml.j2`) and the certificate key in two; the real mitigation is not leaving it running, and nothing needs it after a build. See [Security Posture](). # Design Decisions The rest of this section describes *what* the cluster is. This page covers *why*, including what each choice gives up. Every one has a reasonable alternative; none is the only right answer. ## Flatcar Container Linux, not Talos or a general-purpose distro Flatcar is an immutable, minimal, container-focused OS with A/B updates and declarative first-boot provisioning through Ignition. Nothing is configured by hand on a node, so a node is disposable and reproducible from `inventory.yaml`. Talos goes further — no SSH, no shell, an API-driven machine config — and would remove a whole class of drift. Flatcar keeps a conventional Linux underneath: `ssh`, `systemd`, `journalctl` and `kubeadm` work the way the upstream Kubernetes documentation assumes, which matters more for a cluster that is also a learning environment than the extra hardening does. The cost is that Flatcar's read-only `/usr` forces everything unusual into [sysexts](), Kubernetes included, which is the source of the update behaviour in [Updates & Upgrades](). ## kubeadm, not k3s or a managed distribution kubeadm produces a stock upstream cluster: real etcd, standard control-plane components, and a topology that matches the Kubernetes documentation. k3s would have been far less work — a single binary, batteries included — at the price of a bundled, non-standard set of components. The cost is that everything above the API server is this project's problem: CNI, ingress, storage and certificates are all installed and sequenced explicitly. Most of `payload/` exists because of this choice. ## Cilium as CNI, replacing kube-proxy Cilium replaces `kube-proxy` with eBPF, removing the iptables and IPVS service-routing path. It also supplies Gateway API, L2 announcements for LoadBalancer addresses, WireGuard transparent encryption and Hubble for flow visibility — four things that would otherwise be four components on bare metal. The cost is a hard bootstrap dependency: with no `kube-proxy`, Cilium needs a literal API server address in `k8sServiceHost`, which is the [control plane VIP](); pointing it at one node would make that node a single point of failure for the CNI on every other node, and pointing it at an address that does not answer takes networking down everywhere. ## Gateway API, not Ingress Ingress is effectively frozen, and its per-controller annotations are why Ingress manifests are rarely portable. Gateway API separates the cluster-owned `Gateway` from the app-owned `HTTPRoute`, which fits the split between this repository and the [workloads repository]() exactly: the `Gateway` is platform, the `HTTPRoute` ships with the application. The cost is a smaller ecosystem and more moving parts: CRDs must be installed before anything references them, which is why they are their own Applications that every consumer's sync retries against. ## Rook-Ceph, not Longhorn or local volumes Every worker contributes a raw partition (the control planes keep their disk for etcd, see [Disks]()), and Ceph turns them into replicated block storage that survives a node failure. Local `hostPath` volumes would be simpler and faster, but a node reboot would take its workloads' data with it, and reboots are routine here because that is [how updates get applied](). Longhorn is the closer alternative and easier to operate; Ceph was chosen for its maturity and because the same cluster can later serve object and file storage. The cost: Ceph is the heaviest component in the cluster, wants at least three nodes, and has its own failure modes and vocabulary. It also only provides `ReadWriteOnce` here, since CephFS is not deployed. ## OpenBao, not sealed-secrets or SOPS Sealed-secrets and SOPS keep encrypted material in Git, so rotation is a commit and revocation is impossible after the fact — the ciphertext is in every clone, forever. OpenBao keeps secrets out of the repository and hands them to workloads as ordinary `Secret` objects through the External Secrets Operator. The cost is a manual unseal after every restart, taken deliberately over an auto-unseal dependency outside the house — see [Unsealing after a restart](). ## ArgoCD with an ApplicationSet, not Flux Either would work. ArgoCD was chosen mainly for its UI, which makes sync state and drift legible at a glance — worth more in a homelab, where the operator is often re-learning the system after months away, than Flux's smaller footprint. Bootstrap is two `kubectl apply`s — the AppProjects and the self-managing `argocd` Application — and everything else is discovered from the repository by an ApplicationSet. See [GitOps Strategy](). ## CloudNativePG for every database, not the chart's bundled one Accepting each chart's PostgreSQL subchart is how a cluster ends up running four Postgres versions from four maintainers, each upgraded on someone else's schedule. The rule in the [workloads repository]() is absolute: the subchart is disabled and the application points at a CloudNativePG `Cluster` in its own namespace. The operator generates the credentials into a Secret the chart consumes, so no database password is written down — not in Git, not in OpenBao. The cost is a controller and a CRD to learn, and a major-version upgrade that is explicitly this project's problem rather than something arriving silently in a chart bump — see [CloudNativePG](). ## Trivy Operator, not Kubescape Trivy Operator stores every finding as a plain CRD and scans with the same Trivy that CI uses, so a number on the dashboard can be checked against an object with `kubectl`. Kubescape put results behind an aggregated API server on its own volume, silently reported zero findings for images whose SBOM exceeded a size ceiling, and needed its scanner image pinned ahead of the chart before scheduled scans ran. What was lost is relevancy: Kubescape's eBPF agent could mark a finding as loaded at runtime, and that argument now has to be made from the workload's configuration. # GitOps Strategy ArgoCD manages the cluster state declaratively: if it is not in Git, it is not in the cluster, and if you put it there anyway the next reconcile puts it back. ## Structure ``` flowchart LR subgraph "Bootstrap (once, by hand)" PJ[payload/platform/argocd-projects/projects.yaml] AA[payload/argocd/application.yaml] end AA --> AR[Application argocd] AR --> |"argo-cd chart + values.yaml"| SELF[ArgoCD itself] AR --> |"payload/argocd/applicationset-platform.yaml"| AS[ApplicationSet platform] AS --> |"payload/platform/*/application.yaml"| INFRA[Every component] AS --> |"payload/workloads/application.yaml"| WA[Application workloads] WA --> |"payload/workloads/applicationset.yaml"| APPS[ApplicationSet apps] APPS --> |"homelab-apps: */application.yaml"| WL[Every workload] ``` `argocd` is the only Application not generated by the ApplicationSet and the only one applied by hand. It syncs the chart that runs ArgoCD and, through a second source, the `platform` ApplicationSet, so changes to the generator arrive through Git. Its chart values are described in [Platform → ArgoCD](), along with the recovery commands for the day ArgoCD syncs a broken ArgoCD config. Everything else is a component directory with one `application.yaml`, including ArgoCD's own supporting resources: | Application | Holds | | --- | --- | | `argocd-projects` | the `apps`, `infra` and `system` AppProjects, and a locked `default` so an Application that names no project can do nothing | | `argocd-config` | the `ExternalSecret` for ArgoCD's OIDC client, and its Grafana dashboard; the routes are the chart's, in `payload/argocd/values.yaml` | The AppProjects cannot arrive through ArgoCD alone: an Application naming a project that does not exist is rejected. `payload/platform/argocd-projects/projects.yaml` is applied once at bootstrap, ahead of the `argocd` Application that names `system`, and `argocd-projects` adopts it on its first sync. A broken project definition is fixed with `kubectl apply`, not a sync. Each `application.yaml` is a complete Application, not a fragment: the generator copies its labels, annotations, finalizers and `spec` unchanged, and Renovate and the `Makefile` read it like any other manifest. ## Sync policy Every Application, generated or not, carries the same `syncPolicy`: automated sync with `prune` and `selfHeal`, and a `retry` block of ten attempts with a backoff from 30s to 5m. That is the whole delivery contract: - **A commit is live within one polling interval.** ArgoCD runs on a private address, so no GitHub webhook can reach it; the intervals are in [Platform → ArgoCD](). - **Drift is reverted.** A resource edited or deleted by hand is put back on the next reconcile, in the platform as much as in the workloads. - **A failed sync retries itself.** ArgoCD never re-attempts a failed sync of the same revision unless the Application has a `retry` block, which is why every `application.yaml` carries one. The ten attempts span about forty minutes; after that the Application stays OutOfSync until its revision changes or it is synced by hand: ```bash argocd app sync ``` - **Patching a generated Application's `spec` achieves nothing.** The next ApplicationSet reconcile copies the file back over it. That includes switching automated sync off: the only rollback is a revert commit — see [Rolling back a bad sync](). - **One exception to `prune`, and one annotation.** `kube-vip` syncs with `prune: false`, because its objects are the only thing holding the API VIP; copy the `syncPolicy` from any other sibling. Elsewhere a resource whose deletion would take more than itself carries a per-resource `argocd.argoproj.io/sync-options: Prune=false` instead: a Namespace (everything in it), an AppProject (every Application in it is orphaned), a CloudNativePG `Cluster` and the bucket its backups land in. A removed policy or manifest still prunes; the container it lived in stays. ## Bootstrap convergence Nothing orders the Applications. On a fresh cluster the ApplicationSet generates all of them at once and they converge on their own: a sync that fails because a kind does not exist yet is retried once the chart that ships it has landed, and a resource that applies but cannot go Ready waits without failing anything. What each component waits for is in [Platform → Rollout order](). | While it is missing | The dependent Application | Resolves when | | --- | --- | --- | | A CRD (`ServiceMonitor`, `HTTPRoute`, `ExternalSecret`, a Rook or CNPG kind) | fails its sync and retries | the chart shipping the CRD has synced | | A `StorageClass` | is Progressing, its pods Pending | `rook-ceph-cluster` reports Ready | | A working `ClusterSecretStore` | is Degraded on every `ExternalSecret` | OpenBao is unsealed and holds the key | | A Gateway to attach to | is Progressing on every `HTTPRoute` | `gateway-api` holds its certificates | Two placement rules keep this convergence short. **A resource lives with what it needs, not with what it configures:** the `ClusterSecretStore` is in `openbao`, not `external-secrets`, and cert-manager's issuers and certificates are their own `certificates` Application, so the controllers report Healthy on their own and the resources that wait for OpenBao are the ones that say so. **A resource that fails to apply for longer than the retry budget belongs elsewhere:** the budget covers a chart arriving minutes late, not a human arriving hours late. ### Bootstrap pauses at OpenBao A fresh cluster looks stuck at OpenBao, and that is the design. OpenBao's pods report Ready while sealed, but the `ClusterSecretStore` beside it stays Degraded until OpenBao is initialised, unsealed and has the Kubernetes auth role, and with it every `ExternalSecret` in the platform. `make bao-init` clears that, and `make bao-unseal` after every later restart; `certificates` then waits for `make bao-secrets` to write the Route53 credentials, and the Gateways for the certificates. Nothing fails and nothing times out: every one of those resources applied cleanly and is waiting on status. ## Workloads live in a second repository The applications the cluster exists to run are in [`JanWelker/homelab-apps`](), one directory per application; the `apps` ApplicationSet generates an Application from each `*/application.yaml` there. The split keeps a platform change with a cluster-wide blast radius apart from a workload change with a one-app one, and keeps the reusable half of the cluster in a repository that could build somebody else's. The dependency runs one way: workloads reference the platform (the `apps-gateway`, the `rook-ceph-block` StorageClass, the `openbao` `ClusterSecretStore`, the CloudNativePG operator), and the platform references the workloads repository exactly once — the `repoURL` in `payload/workloads/applicationset.yaml`. ### What the split costs - **Two repositories to clone**, and a change that spans both is two pull requests that cannot merge atomically; merge the platform half first. - **The diff preview covers only this repository.** Both ApplicationSets carry `argocd-diff-preview/ignore` because the tool renders the repository it is given. - **A workload can reference a platform resource that does not exist**, and nothing catches it until ArgoCD tries to sync. ### Workloads sync alongside the platform The `apps` ApplicationSet ships inside the `workloads` Application, which the `platform` ApplicationSet generates like any component. That is why `payload/workloads/` is a sibling of `payload/platform/`, and why the `platform` ApplicationSet names it as a second generator path rather than a glob: ```yaml files: - path: payload/platform/*/application.yaml - path: payload/workloads/application.yaml ``` On a fresh cluster the workloads therefore start syncing before the platform under them is Healthy, and converge the same way it does: a workload `application.yaml` carries the same `retry` block, its `HTTPRoute` waits for the `apps-gateway`, its volumes for Rook, its secrets for OpenBao. Ordering *within* a workload uses resource-level sync waves — the CloudNativePG `Cluster` is in an earlier wave, and ArgoCD's built-in health check for `postgresql.cnpg.io/Cluster` waits for it before applying the application. ## Moving a resource to another Application Moving a manifest between directories is not a move to ArgoCD: the old Application sees a tracked resource vanish and, with `prune: true`, deletes it; the new one recreates it, possibly seconds later. For a `ClusterSecretStore` that is an outage of every secret; for a child `Application` with the resources finalizer it deletes everything that Application deployed. A move takes two merges: 1. Annotate the resource where it is, and let that sync: ```yaml annotations: argocd.argoproj.io/sync-options: Prune=false argocd.argoproj.io/compare-options: IgnoreExtraneous ``` Both are read off the **live** object. `Prune=false` stops whichever Application prunes first; `IgnoreExtraneous` keeps the old Application Synced while it still sees the resource. 2. Move the file, keeping the annotations. The new Application takes over the tracking annotation. The annotations can go once the second change has synced everywhere. ## Version pins Each chart version is pinned once, in the `application.yaml` that owns the component; the node-level versions (Flatcar, Kubernetes, syslinux) are pinned once in `ansible/inventory.yaml`. The `Makefile` reads `targetRevision` out of the manifests for the bootstrap installs (`make install-cilium`, `make install-argo`), so what bootstrap installs is what ArgoCD then reconciles. kube-vip is the one double pin: `kube_vip_version` in the inventory for the bootstrap static pod, and the image in `payload/platform/kube-vip/daemonset.yaml` for what runs after the first sync. Renovate tracks all of them; its automerge policy is in [Maintenance](). # Known Limitations Things this cluster does not do, collected so they are findable before they are discovered. None is a bug; each is an accepted tradeoff or work not yet done. | Limitation | Consequence | Mitigation / plan | Where documented | | --- | --- | --- | --- | | **The API endpoint address is fixed at `kubeadm init`** | `controlPlaneEndpoint` is the kube-vip VIP, and `kubeadm` bakes it into the API server certificates, every node's kubeconfig and the stored `ClusterConfiguration`; Cilium's `k8sServiceHost` names the same address. Moving the VIP to another address is therefore a certificate regeneration on every control-plane node and a kubeconfig rewrite on every node, not a config change that can be synced | Choose the address once, outside any DHCP range and apart from the LoadBalancer pools. The in-place procedure exists for the day it has to move; the next build [uses a name]() | [Choosing the address](), [migrating in place]() | | **OpenBao needs an operator to unseal it** | Shamir seal, no auto-unseal: every restart (node reboot, Kured cycle, chart bump) leaves each replica sealed until the [threshold of key shares]() is supplied. Sealed, no `ExternalSecret` resolves and cert-manager cannot renew. The cluster looks healthy; the consequence surfaces when a certificate expires | Check `bao status` after every reboot. Auto-unseal against a KMS or a second OpenBao would remove the step at the price of an external hard dependency; the key shares stay in the operator's hands on purpose | [Unsealing after a restart](), [after any node reboot]() | | **Alerting reaches one mailbox** | One receiver, one mailbox, one SMTP provider, unmonitored. If mail is rejected or the password expires, alerts stop silently; `Watchdog` proves the pipeline only as far as Alertmanager | A second receiver on a different transport. Meanwhile, run the routine health check | [Alerting](), [routine health check]() | | **Backups do not leave the cluster** | Velero and the etcd snapshot CronJob write to the Ceph object store the cluster runs on: protection against a deleted PVC or a bad `prune`, none against losing the cluster | RGW bucket replication or a second Velero `BackupStorageLocation`; neither is configured. Until then disaster recovery is Git plus the OpenBao unseal keys | [Backups & Recovery]() | | **Automatic updates stop at patch releases** | Sysupdate is pinned to the Kubernetes major.minor in `ansible/inventory.yaml`; Kured drains and reboots one node at a time to apply what is staged, and refuses while Ceph or etcd is unhealthy. A minor Kubernetes upgrade is manual: change the inventory, push the sysupdate config to running nodes, run `kubeadm upgrade`. Kured blocks only on the alerts named in its config; anything outside that list does not stop a reboot | By design | [Nodes are pinned to a minor series](), [upgrading a minor deliberately](), [Kured]() | | **The `infra` AppProject is permissive** | The network policies are enforced, but the `infra` AppProject allows every kind in every namespace, so a platform `application.yaml` can deploy anything anywhere | Narrow the projects | [Security Posture]() | | **Reprovisioning needs the boot server on the segment** | Rebuilding a node means `make serve` on a machine on the nodes' L2 segment, the external DHCP pointing at it, `make reinstall LIMIT=`, and a power cycle. There is no remote rebuild, and the deployment host is not part of the cluster | Only *re*provisioning: Flatcar is installed to disk, so a running node reboots, updates and rejoins with the boot server off | [Boot & Bootstrap Process]() | | **The object store survives one host down, no more** | During every Kured reboot the RGW data pool, which holds the Velero backups and Loki chunks, has no redundancy left, while `rbd-pool` still has some | Accepted, no migration: Ceph cannot change the erasure profile of an existing pool, so changing it means a new pool and moving every bucket | [Object storage]() | | **Single region, site and rack** | No failure domain larger than a node. Ceph replicates across nodes on one power feed; a site-level event takes everything | The honest limit of a homelab | — | # Security Posture This is a homelab on a private network, and several deliberate shortcuts follow from that. They are listed so that someone reading the manifests can tell a decision from an oversight. ## Assumed trust boundary The cluster assumes a **trusted L2 network segment**: anything with a port on it is treated as friendly. There is no VPN requirement and no mutual TLS between components; inside the cluster, the default-deny network policies in [Authorization](<#authorization>) are the segmentation. The published hostnames are a partial exception. Certificates come from Let's Encrypt through a DNS-01 challenge against a public zone, so `argo.infra.k8s.wlkr.ch` and its siblings are publicly resolvable and appear in Certificate Transparency logs, even though they point at RFC1918 addresses. Plan names accordingly. ## Provisioning Provisioning is the least protected phase, by design: it has to work before any of the cluster's own security exists. | Property | Detail | | --- | --- | | Ignition configs are served unauthenticated over HTTP | Anything on the segment can fetch `http://:8000/ignition-.json` while the boot server is running | | Those configs embed join credentials | The bootstrap `token`, and on control-plane nodes the `certificateKey`, together enough to join a new control-plane node. Only the init host has them in its kubeadm config; the others get a `0600` environment file that `bootstrap-k8s.service` names by variable and deletes after a successful join, so `systemctl cat` and `systemctl show` reveal nothing | | The control-plane configs embed the etcd encryption key | `/etc/kubernetes/enc/encryption-config.yaml` is inlined with the key that encrypts every Secret at rest. It has no TTL: anyone who fetched a control-plane config while the boot server was up can read Secrets out of any etcd backup, for the life of the cluster | | Nodes join with `--discovery-token-unsafe-skip-ca-verification` | A joining node does not verify the API server's CA | | Sysext updates set `Verify=false` | On first boot Ignition checks each image against the sha256 the boot server was given, so a substituted image fails. Later sysupdate downloads travel over HTTPS with signature verification off. The transfer config that names their source is inlined from a template in Git, not fetched | | The PXE kernel, initrd and bootloader are signature-checked | `make download` takes their sums only from a clearsigned file that `gpgv` verifies against a pinned primary key fingerprint: Flatcar's image signing key for `.DIGESTS.asc`, the kernel.org checksum autosigner for syslinux's `sha256sums.asc` | | The OS image is signature-checked | `flatcar-install` is given `-b`/`-V`, so the node downloads the image and its detached signature and checks both against Flatcar's key before writing anything | The mitigation is time: `make serve` is a foreground command, the bootstrap token and the certificate key expire (the windows are in [Switching back to local boot]()), and `make config` re-arms both only for those windows. The encryption key has no clock, which is why the HTTP server listens only on `boot_server_ip`, serves files by name and lists nothing. TFTP listens on every interface, because macOS lets an unprivileged process bind port 69 only there, and refuses every write request, so nobody can plant a boot menu. **Stop the boot server when provisioning is finished.** Flatcar is installed to disk, so nothing needs it after a build — see [Every boot after the first](). `output/credentials/` holds the generated bootstrap token and certificate key in plaintext. The directory is `0700`, the Ignition configs in `output/http` are `0600`, and `output/` is gitignored. The values are reused across `make config` runs, because a node whose join window passed retries with the values on its disk once `make config` has re-armed them ([Adding a node]()). Delete them to force new ones. The installer keeps its copy of the node config `0600` in RAM; `flatcar-install` carries that mode to the OEM partition. ## Node hardening `ansible/templates/kubeadm.yaml.j2` and `ansible/templates/butane_node_config.yaml.j2` carry the settings below. They land at provisioning time and change only on `make reinstall`. | Setting | Why | | --- | --- | | `proxy.disabled: true` | Cilium replaces `kube-proxy`. The field lands in the `kubeadm-config` ConfigMap that `kubeadm upgrade apply` reads; a `skipPhases` on init alone leaves `proxy: {}` stored, and every upgrade puts `kube-proxy` back beside Cilium | | `--kubelet-certificate-authority` | The API server verifies the kubelet's serving certificate for exec, logs, attach and port-forward, the calls that carry command output. It can, because `serverTLSBootstrap` makes the kubelet request its certificate from the cluster CA and [kubelet-csr-approver]() approves the CSR; metrics-server verifies the same certificates | | `--admission-control-config-file` | A [Pod Security default]() for namespaces without labels, so a namespace nobody labelled is `baseline`, not `privileged` | | `--profiling=false` on the API server, controller-manager and scheduler | `/debug/pprof` exposes internals to any token with the right RBAC and helps nothing here; CIS asks for it off | | Metrics listeners on the node address | etcd's plain-HTTP metrics and the controller-manager and scheduler bind the node's own address, where Prometheus scrapes them, instead of every interface. The address differs per node, so it comes from per-node JSON patches in `/etc/kubernetes/patches` rather than the cluster-wide `ClusterConfiguration`; the patches also move the probes, and etcd keeps `127.0.0.1` for its own | | `DenyServiceExternalIPs` | `spec.externalIPs` lets whoever can create a Service claim traffic to any address on every node, with no ownership check. Nothing here uses it: LoadBalancer addresses come from Cilium's pools. `NodeRestriction` is repeated because the flag replaces kubeadm's default list | | `--tls-cipher-suites` and `--tls-min-version`, `tlsCipherSuites` and `tlsMinVersion` | Go's defaults still include CBC and 3DES suites; the list keeps AEAD suites on ECDHE, which every client here speaks (kubectl, the controllers and the kubelets are all Go). TLS 1.3 ignores the list; the floor stops a downgrade to 1.0 or 1.1. The kubelet's own listener on port 10250 gets the same list and floor, since the API server, metrics-server and Prometheus are its only clients | | `shutdownGracePeriod` and `shutdownGracePeriodCriticalPods` | The kubelet holds a systemd inhibitor lock, so a `systemctl reboot` without the [documented drain](), or an ACPI power-off, waits while pods get their SIGTERM window instead of killing them mid-write; the second value is the tail of the first that only critical pods may use. CloudNativePG primaries and Ceph OSDs are the ones that notice | | `protectKernelDefaults: true` | The kubelet no longer rewrites `vm.overcommit_memory`, `vm.panic_on_oom`, `kernel.panic` and `kernel.panic_on_oops` at start and refuses to start unless they are already right. `/etc/sysctl.d/k8s.conf` in the node config sets them, and the two travel together: without the drop-in a fresh node comes up at the kernel defaults, the kubelet exits and the node never joins, which shows at the next `make reinstall`, not on merge | | `kernel.panic` with `kernel.panic_on_oops` | An oops reboots the node after a few seconds instead of leaving it wedged, so Ceph and etcd start recovering without a hand power-cycle | | `podPidsLimit` | Bounds the PIDs one pod can hold, so a process that stops reaping its children cannot exhaust the node's PID space and take the kubelet with it. Well above anything running here, well below `kernel.pid_max` | | `systemReserved`, `kubeReserved` and `evictionHard` memory | The reservations cap `kubepods.slice` below the node's RAM, so a pod that grows, the API server included, is OOM-killed inside its own cgroup instead of starving systemd-networkd, sshd and the kubelet. The eviction threshold makes the kubelet evict before the kernel has to act. A running node needs the same values in the `kubelet-config` ConfigMap, or `kubeadm upgrade node` reverts them | | `/etc/tmpfiles.d/kubernetes-cis.conf` | kubeadm writes its files 0644 and Cilium creates the CNI directory 0755; CIS wants 0600 and 0700. systemd-tmpfiles applies the modes at boot and a kubelet `ExecStartPre` re-applies them before every start, when kubeadm has just rewritten them. See [Triage 2026-09-20]() | | `systemd-sysupdate-reboot.timer` masked | Flatcar ships a vendor symlink that runs it despite a disabled preset. It would reboot on its own schedule, behind [Kured]() and without a drain, and it fails every run anyway: it acts on the default sysupdate component, which holds only `noop.conf` | | `noop.conf` in `/etc/sysupdate.d/` | The unsuffixed directory is the default component that `systemd-sysupdate.service` acts on; the no-op transfer keeps that run from failing while the real transfer sits in the `kubernetes.d` component | ## Secrets Secrets live in [OpenBao]() and reach workloads as native Kubernetes `Secret` objects through the [External Secrets Operator](). Nothing sensitive is committed to Git. - OpenBao is sealed with Shamir and [unsealed by hand](), so the 5 key shares are the root of trust for every other secret; losing all of them loses everything, and nothing outside the cluster holds a copy. - No root token is kept: `make bao-init` revokes it once it has created the [operator login](), and one is generated from the shares when needed. The operator login reads and writes every KV path, so `output/credentials/` is as sensitive as before, but it cannot change policies or auth methods. - A workload's namespace reaches only `kv//` in OpenBao; the platform's store refuses every other namespace — see [Kubernetes auth](). - A Kubernetes `Secret` is base64, not encryption. Anyone with `get secrets` in a namespace can read what ESO materialised there. Encryption at rest protects the bytes in etcd, not the API. - Velero's resource backups hold every `Secret` as plain JSON in the `velero` bucket, so its keys and RGW bypass encryption at rest; only the volume data is encrypted — see [Velero](). ### Encryption at rest The API server encrypts `secrets` with `secretbox` before they reach etcd (`ansible/templates/kubeadm.yaml.j2`; the key file in `ansible/templates/butane_node_config.yaml.j2`). Without it an etcd backup, a stolen disk, or read access to `/var/lib/etcd` yields every credential the cluster holds. | Property | Detail | | --- | --- | | Provider | `secretbox`, with `identity` listed after it | | Key | 32 random letters and digits (about 190 bits), generated once by `make config` into `output/credentials/encryption_key` and base64-encoded into the config as the `secretbox` key | | Scope | `secrets` only; ConfigMaps and other resources are unencrypted | | Distribution | The same key on every control-plane node, written by Ignition to `/etc/kubernetes/enc/encryption-config.yaml` (mode `0600`) | With `identity` last, new writes are encrypted and Secrets written before it was enabled stay readable but are not rewritten. To encrypt what already exists, once the API servers have restarted: ```bash kubectl get secrets -A -o json | kubectl replace -f - ``` The key lives in `output/credentials/` beside the kubeadm token and certificate key — that directory is what decrypts etcd. The API servers reload the config file on change (`encryption-provider-config-automatic-reload`), so rotating it is a file edit and a rewrite of every Secret, by hand — see [Encryption Key Rotation](). A KMS provider would remove the key file at the cost of a dependency the cluster must reach before it can serve Secrets. ## Runtime [Tetragon]() records every exec in every pod and alerts when anything but the control plane reads the cluster's key material. It is detection, not prevention: nothing is killed, and a policy that fails to load on a new kernel watches nothing until its alert fires. ## Audit logging The API server records who did what, to which object, and whether it was allowed; the `audit` half of the Pod Security Admission labels has nowhere to go without it. Policy, retention and queries are in [Audit Logging](). ## Authorization **AppProjects constrain the workloads, not the platform.** `payload/platform/argocd-projects/projects.yaml` defines them. `infra` allows `sourceRepos: "*"` and every group and kind in every namespace: it is this repository deploying this repository, and a restriction there guards against nothing the PR review does not. `system` is confined to the `argocd` namespace. `apps` has the boundary to draw, because its Applications come from a second repository with a lighter review: it accepts only the workloads repository and the chart repositories the workloads use, may not write into any platform namespace (`authentik` excepted, where an admission policy narrows it to the workload's own blueprint ConfigMap), and at cluster scope may create nothing but namespaces, never at the `privileged` Pod Security level — see [Security Policies](). **Every platform namespace is default-deny in both directions**, one `CiliumNetworkPolicy` file per namespace plus the cluster-wide rules, and Cilium enforces them: `policyAuditMode` is off, so a flow no rule allows is dropped — see [Security Policies]() and [Rollout](). **Pod Security Admission is on, but mostly auditing.** Every platform namespace carries `enforce` at the level it demonstrably needs and `warn`/`audit` at a stricter one, so violations are visible without breaking what runs: measure first, enforce second. **Image references are checked at admission.** An untagged or `latest` image is refused; a registry outside the allowlist is recorded and warned about but admitted. Signatures are not verified: few of the upstreams sign, so a verifier would exempt nearly everything. See [Security Policies](). **Both Gateways admit routes from every namespace** (`allowedRoutes.namespaces.from: All`). Any namespace can attach an `HTTPRoute` to either Gateway and claim a hostname inside its listener's pattern — see [Gateway API](). ## Exposed interfaces Every platform UI on the infra gateway, and Flowscape and the Extractor admin UI on the apps gateway, is behind [Authentik](), and only accounts in `infra-users` get past it. The wollbi editors also have Authentik accounts, so being logged in is not enough. The last two rows are not behind Authentik: | Service | Authentication | Group required | | --- | --- | --- | | ArgoCD | Authentik OIDC; local admin disabled. The server runs with `--insecure` because TLS terminates at the Gateway | `infra-users`, then `argocd-admins` or `argocd-viewers` for any access | | Grafana | Authentik OIDC; login form disabled | `infra-users`; refused without `grafana-admins`, `grafana-editors` or direct `infra-users` | | Hubble UI | Authentik proxy outpost | `infra-users` | | Rook dashboard | Authentik proxy outpost | `infra-users` | | Prometheus | Authentik proxy outpost | `infra-users` | | Alertmanager | Authentik proxy outpost | `infra-users` | | Flowscape, Extractor admin UI | Authentik proxy outpost | `infra-users` | | OpenBao UI | Token or configured auth method — not behind Authentik | — | | Agent SSH, `*.ssh.wlkr.ch` | The `ssh` TLS-passthrough listener on `apps-gateway`, which external-dns publishes on the public address. Nothing in front authenticates it: `sshd` in each agent pod admits the user `agent` with public keys only — see [SSH listener]() and the workloads repository's [Claude agents]() page | — | The groups and how they nest are in [Groups and roles](). Authentik is therefore a dependency of reaching any of them; the break-glass paths are in [When Authentik is down](). OpenBao is left out deliberately: it holds Authentik's own database password, so putting it behind Authentik would be a loop. ## What would tighten this up Roughly in order of value against effort: 1. Narrow `sourceRepos` on the AppProjects to this repository and the Helm repositories actually in use. 2. Restrict `allowedRoutes` on `infra-gateway` to the platform namespaces, and promote the registry allowlist to `Deny`. 3. Add a second Alertmanager receiver on a different transport, so a failure of the mail path is not itself unmonitored. 4. Move etcd encryption to a KMS provider, removing the static `encryption_key` in `output/credentials/`. 5. Get the backups out of the cluster: Velero and the etcd snapshots write to the Ceph object store they are backing up. See [Known Limitations]() for the operational counterparts. # Platform # Platform The core infrastructure components that run the cluster, all managed by ArgoCD. Every one of them exists because bare metal does not come with the thing a cloud provider would have handed you: no load balancer, no managed certificates, no block storage API, no identity provider, no backup service. ## Components One row per directory under `payload/platform/`, with the Application name where it differs from the directory — that is the name `kubectl -n argocd get applications` shows — and the namespace it lands in. | Component | Namespace | What it does | | --- | --- | --- | | alloy | `logging` | Grafana Alloy, the log shipper — see [Logging]() | | arc-controller | `arc-systems` | The Actions Runner Controller — see [GitHub runners]() | | [arc-runners]() | `arc-runners`, `arc-jobs` | Self-hosted GitHub Actions runners, one scale set per repository | | [argocd]() | `argocd` | ArgoCD itself and the `platform` ApplicationSet | | argocd-config | `argocd` | The `ExternalSecret` for ArgoCD's OIDC client, and its Grafana dashboard; see [ArgoCD]() | | argocd-projects | `argocd` | The AppProjects; see [GitOps Strategy]() | | [authentik]() | `authentik` | Single sign-on for every platform UI | | backup | `backup` | Velero's buckets, the `VolumeSnapshotClass` and an etcd snapshot CronJob — see [Backups & Recovery]() | | [cert-manager]() | `cert-manager` | The cert-manager chart | | certificates | `cert-manager` | The Let's Encrypt issuers and the Gateway wildcard certificates — see [cert-manager]() | | [cilium]() | `kube-system` | CNI, `kube-proxy` replacement, Gateway API, LoadBalancer addresses, WireGuard, Hubble | | [cloudnative-pg]() | `cnpg-system` | The PostgreSQL operator every workload database runs on | | [descheduler]() | `descheduler` | Evicts pods a drain or reboot left piled on the surviving nodes | | [external-dns]() | `external-dns` | Publishes Route53 records from HTTPRoutes | | [external-secrets]() | `external-secrets` | Bridges OpenBao to native Kubernetes Secrets | | [gateway-api]() | `kube-system` | The two Gateways and the HTTP-to-HTTPS redirect | | gateway-api-crds | `kube-system` | The Gateway API CRDs — see [Gateway API]() | | kube-vip | `kube-system` | Holds the control-plane VIP; adopts the static pod Ignition bootstraps — see [Control Plane VIP]() | | kubelet-csr-approver | `kubelet-csr-approver` | Approves `kubelet-serving` CSRs against the inventory — see [Metrics Server]() | | [kured]() | `kured` | Drains and reboots nodes to apply staged OS and Kubernetes updates | | [logging]() | `logging` | Loki's bucket, rules, dashboards and Grafana datasource | | loki | `logging` | The Loki chart — see [Logging]() | | [metrics-server]() | `kube-system` | The `metrics.k8s.io` API behind `kubectl top` and every HPA | | monitoring, Application `kube-prometheus-stack` | `monitoring`, `node-exporter` | Prometheus, Grafana, Alertmanager, kube-state-metrics; node-exporter alone in `node-exporter` — see [Monitoring]() | | [openbao]() | `openbao` | Cluster-wide secret store | | prometheus-operator-crds | `monitoring` | The Prometheus operator CRDs, separate so every chart that ships a `ServiceMonitor` can sync — see [Monitoring]() | | [rook-ceph]() | `rook-ceph` | The CSI driver, dashboards and HTTPRoute | | rook-ceph-cluster | `rook-ceph` | The `CephCluster`, pools and storage classes — see [Rook-Ceph]() | | rook-ceph-operator | `rook-ceph` | The Rook operator chart — see [Rook-Ceph]() | | security | `kube-system` | Pod Security Admission levels, the network policies and image admission policies — see [Security Policies]() | | snapshot-controller | `backup` | The CSI snapshot controller Velero's volume snapshots need — see [Backups & Recovery]() | | [tetragon]() | `kube-system` | Runtime detection: execs, credential changes and sensitive file reads from the kernel | | [trivy-operator]() | `trivy-system` | Vulnerability, misconfiguration, secret, RBAC and CIS scanning, every finding a CRD | | [velero]() | `backup` | Volume and resource backups to the Ceph object store | Across the platform, memory limits are set at roughly 2.5x the measured peak working set and requests at steady state; CPU is requested but never limited. The control-plane static pods get their memory requests through the kubeadm patches in `ansible/templates/butane_node_config.yaml.j2`, applied at `kubeadm init`, `join` and `upgrade`, so a running node picks them up at its next reinstall; they carry no limits, because a limit on the API server or etcd would OOM-kill the control plane and 2.5x their peak exceeds the node. CoreDNS is kubeadm's too: the same directory holds a patch spreading its replicas one per node, which kubeadm applies at `init` and `upgrade apply` only, and its PodDisruptionBudget sits in the `security` component because nothing else owns a CoreDNS manifest. Components the cluster cannot run without — the CNI and API VIP, the admission webhooks, the secret and certificate chain, the CSR approver, the storage daemons and the node security agents — carry a `system-*` priority class, so under node pressure the kubelet evicts workloads first: `system-node-critical` for the DaemonSets a node needs, `system-cluster-critical` for the rest. ## Traffic Flow ``` flowchart LR Client([Client]) --> LB[Cilium LoadBalancer] LB --> GW[Gateway API] subgraph Cluster GW -->|HTTPRoute| Svc[Service] Svc --> Pod[App Pod] end style Client fill:#f9f,stroke:#333 ``` Four hops, and Cilium is three of them. When a hostname stops answering, ask which hop stopped, in this order: does the Gateway still hold its LoadBalancer IP, does the `HTTPRoute` still say `Accepted`, does the Service still have endpoints. ## HTTPRoute Locations Routes sit with the component they expose, as a manifest where the component is plain YAML and as chart values where the chart renders its own: | Service | URL | Route | | --- | --- | --- | | ArgoCD | `argo.infra.k8s.wlkr.ch` | `server.httproute` in `payload/argocd/values.yaml` | | ArgoCD CLI (gRPC) | `argo-grpc.infra.k8s.wlkr.ch` | `server.grpcroute` in `payload/argocd/values.yaml` | | Authentik | `auth.k8s.wlkr.ch` | `payload/platform/authentik/httproute.yaml` | | Prometheus | `prometheus.infra.k8s.wlkr.ch` | `payload/platform/monitoring/httproute.yaml` | | Alertmanager | `alertmanager.infra.k8s.wlkr.ch` | `payload/platform/monitoring/httproute.yaml` | | Grafana | `monitoring.infra.k8s.wlkr.ch` | `grafana.route.main` in `payload/platform/monitoring/application.yaml` | | Hubble | `hubble.infra.k8s.wlkr.ch` | `payload/platform/cilium/httproute.yaml` | | OpenBao UI | `vault.infra.k8s.wlkr.ch` | `server.gateway.httpRoute` in `payload/platform/openbao/application.yaml` | | Rook dashboard | `rook.infra.k8s.wlkr.ch` | `payload/platform/rook-ceph/httproute.yaml` | | Ceph object store | `s3.infra.k8s.wlkr.ch` | `payload/platform/rook-ceph/httproute.yaml` | | Home Assistant | `home.k8s.wlkr.ch` | `home-assistant/httproute.yaml` in [homelab-apps]() | ## Deployment ```bash make bootstrap # Gateway API CRDs + Cilium, ArgoCD, then the handover ``` After that the `argocd` Application syncs the `platform` ApplicationSet, which generates one Application per `payload/platform/*/application.yaml` plus the `workloads` Application that deploys the `apps` ApplicationSet for the [workloads repository](). Each component directory holds exactly one `application.yaml`; everything else in it is what that Application deploys. [GitOps Strategy]() has the structure and the sync policy. ## Replica placement Two replicas on one node are not redundancy. Every chart's default is a single `preferredDuringSchedulingIgnoredDuringExecution` anti-affinity term — one weight among the scheduler's scores, and on a cluster of one 16 GB worker against two 8 GB ones, free memory outscores it. So every component with more than one replica carries an explicit rule: | Rule | When | | --- | --- | | `topologySpreadConstraints`, `maxSkew: 1`, `whenUnsatisfiable: ScheduleAnyway` | the component runs on the three workers only, where a hard rule would leave a replica `Pending` through a Kured drain | | `podAntiAffinity` `requiredDuringSchedulingIgnoredDuringExecution` | the component tolerates the control planes, so every node is a candidate and a drain always leaves room — or its chart exposes no spread constraint | Constraints carry `nodeTaintsPolicy: Honor`, so the skew is measured over the nodes the pod can reach rather than over the tainted control planes. Where a component carries a PodDisruptionBudget it says `maxUnavailable: 1`, so a drain or the [descheduler]() evicts one replica at a time; `maxUnavailable` rather than `minAvailable`, so a component scaled down to one replica never blocks a drain. The CloudNativePG operator runs two replicas with no budget. Placement is scored once, at scheduling time: pods already sharing a node stay there until something restarts them, and the [descheduler]() moves them back only if its evictor is allowed to touch them. ## Rollout order Nothing enforces an order: every Application syncs as soon as it exists and converges by retry and health, as [GitOps Strategy]() explains. This is the order a fresh cluster settles in, and what each layer is waiting for while it is not yet Healthy. | Layer | Applications | Waits for | | --- | --- | --- | | Projects and CRDs | `argocd-projects`, `gateway-api-crds`, `prometheus-operator-crds` | — | | Network | `cilium`, `kube-vip` | the Gateway API CRDs Cilium's operator reads at startup | | Controllers | `arc-controller`, `cert-manager`, `cloudnative-pg`, `external-secrets`, `kubelet-csr-approver`, `rook-ceph-operator`, `snapshot-controller` | a network; each brings its own CRDs | | Storage | `rook-ceph`, `rook-ceph-cluster` | the Rook operator and its CRDs | | Secrets | `openbao` | `rook-ceph-block` for its volumes, then an operator — see [Bootstrap pauses at OpenBao]() | | Certificates | `certificates`, `external-dns` | a working `ClusterSecretStore` and the Route53 credentials in OpenBao | | Ingress | `gateway-api` | the wildcard certificates the Gateways terminate TLS with | | Services | `arc-runners`, `argocd-config`, `authentik`, `backup`, `kube-prometheus-stack`, `logging`, `metrics-server` | secrets, storage, the Gateways, and approved kubelet certificates | | Backends | `loki`, `velero` | the buckets `logging` and `backup` claim | | Agents | `alloy`, `tetragon` | Loki, so the collector has somewhere to ship | | Policy | `descheduler`, `kured`, `security`, `trivy-operator` | namespaces to label, a Prometheus to scrape and metrics-server for node usage; kured reboots only on a staged update, never on a fresh install | | Workloads | `workloads` | nothing on its own; the workloads it generates wait for the platform resources they reference — see the [workloads repository]() | Why the `ClusterSecretStore` sits with OpenBao and the issuers are their own Application is in [Bootstrap convergence](). # ArgoCD ArgoCD is the delivery layer: everything in `payload/` and in the workloads repository reaches the cluster through it. How the Applications are generated and synced is in [GitOps Strategy](); this page covers the chart values and the supporting resources around ArgoCD itself. ## At a glance | | | | --- | --- | | Namespace | `argocd` | | Depends on | [OpenBao]() for the OIDC client secret, [Gateway API]() for its route, [Authentik]() for every login | | If it is down | Nothing syncs. Running workloads are unaffected | | Health check | `kubectl -n argocd get applications` → every one `Synced` and `Healthy` | | Files | `payload/argocd/` (Application, `platform` ApplicationSet, `values.yaml`), `payload/platform/argocd-config/` (OIDC `ExternalSecret`, Grafana dashboard), `payload/platform/argocd-projects/` (AppProjects) | ## Configuration The parts of `payload/argocd/values.yaml` that are not self-explanatory: | Setting | Why | | --- | --- | | `redis-ha.haproxy` `maxSurge: 0` | Three replicas with hard per-host anti-affinity on three schedulable nodes. The default surges a fourth pod with nowhere to land and wedges every rollout; retiring first frees the node | | `redis-ha.image.tag`, `redis-ha.haproxy.image.tag` | Both run ahead of the chart, whose exact patch tags stop being rebuilt once the next one lands and so miss base-image security fixes. Renovate carries them forward; each bare `tag:` must be listed under the `pinDigests: false` rule in `renovate.json` | | Memory limits, no CPU limits | A CPU limit throttles even on an idle node; memory is not compressible. Sizing is in [Platform]() | | `metrics.enabled` on four components | Creates the `-metrics` Services whose names are the `job` label the vendored dashboard filters on. The ServiceMonitors render only once the Prometheus operator CRDs exist, so `make install-argo` still works first | | `configs.params` `server.insecure: true` | TLS terminates at the Gateway. The chart reads this parameter, not `extraArgs`, to point its route at the plaintext port | | `server.httproute` | The chart renders the `HTTPRoute` to `argocd-server`, so it is not a file of its own; the response-header filters are the same every UI route carries — see [Gateway API]() | | `resource.customizations` for `HTTPRoute` | The API server defaults `group`, `kind`, `weight` and a catch-all `matches` onto every route; without the ignore list each one is permanently OutOfSync. Only the catch-all value is ignored, so a changed path still syncs | | `dex.enabled: false` | `oidc.config` talks to [Authentik]() directly, so the chart's bundled Dex would be a second login service with nothing behind it; the redirect URI is the server's own `/auth/callback` either way | | `admin.enabled: "false"` | With SSO in front, a shared admin password would bypass it with no audit trail | | `resource.customizations.health.argoproj.io_Application` | Restores health assessment for `Application` resources, dropped in ArgoCD 1.8, so `argocd` reports the health of the ApplicationSet rather than a permanent Healthy | | `policy.default: ""` | An authenticated user with no matching Authentik group gets no access, not read-only-everything | | `timeout.reconciliation` and the ApplicationSets' `requeueAfterSeconds` | ArgoCD listens on a private address, so GitHub's webhook cannot reach it and a commit waits for the next poll. The two run separately: the first refreshes Applications, the second re-reads the generator paths. The repo-server caches a revision for the first, so it must not exceed the second or a new commit's files stay invisible to the generator until the cache expires | The OIDC client ID and secret come from `kv/authentik/config`, the same OpenBao path Authentik's blueprint reads. `argocd-cm` refers to them as `$argocd-oidc:client-id`; ArgoCD resolves a `$name:key` reference only against a Secret labelled `app.kubernetes.io/part-of: argocd`, which the `ExternalSecret` template sets. The Grafana dashboard in `grafana-dashboards.yaml` is upstream's `examples/dashboard.json`, unmodified, at the ArgoCD version the chart deploys. Renovate does not see it: re-copy it when ArgoCD moves a minor version. ## Usage ### Projects The `apps` project lists the repositories its Applications may pull from in `payload/platform/argocd-projects/projects.yaml`. An OCI Helm chart is listed and referenced **without** the `oci://` scheme (`ghcr.io/janwelker/charts` as `sourceRepos` entry and as the Application's `repoURL`, with `chart:` naming the chart): Argo CD compares the two strings after normalising, and treats a `repoURL` with no scheme as OCI, so a public registry needs no repository Secret and no `enableOCI`. The `default` project Argo CD creates on install is declared in the same file with empty lists: it cannot be deleted, so an Application that names no project lands there and must be able to do nothing. ### Agent account The `claude` account is the in-cluster Claude Code agents' API identity. It may read everything `role:readonly` reads, `argocd app get --refresh` included, and nothing else: the sessions investigate and open pull requests, and every Application syncs itself. It has `apiKey` only, so it cannot log in. One token serves every session with `argocd.enabled`, so its expiry is what bounds a leak from any of them. The token expires after 90 days: that caps a leak at a quarter, and rotating is a few commands with no restart. An expired token fails every agent `argocd` call until the next one is stored. To rotate: 1. Generate the token, logged in through Authentik: ```bash argocd account generate-token --account claude --expires-in 90d ``` 2. Store it with `make bao-secrets`, at `kv/claude-agents/argocd` (`token`). 3. Pull it into the sessions now instead of at the next `refreshInterval`. The chart mounts the token as a file that the `argocd` wrapper reads on every call, so no pod restarts; only a shell opened earlier keeps the old token in its environment: ```bash kubectl -n claude-agents annotate externalsecret claude--argocd force-sync=$(date +%s) --overwrite ``` 4. Revoke the previous token. `argocd account get --account claude` lists the token IDs with their expiry dates: ```bash argocd account delete-token --account claude ``` The agents reach Argo CD on 443 through the CLI host from [CLI access](<#cli-access>), which resolves to `infra-gateway`'s own address. The caller's policy has to admit both legs, as for any name the Gateway serves ([Security Policies]()): | Leg | Admitted by | | --- | --- | | Agent pod to the Gateway address | A name rule for the CLI host: the datapath classifies the Gateway's address as `world`, not `ingress` | | The Gateway's Envoy to `argocd-server` | An egress rule to the `argocd-server` pod itself. Envoy checks the caller's egress policy again against the pod it picked and answers `403 Access denied` when no rule matches, audit mode or not. The server's own policy already admits the `ingress` entity | The `homelab` session's policy in the workloads repository carries both. The chart's own policy has only the name rule ([claude-agent\#3]()), so any other session with `argocd.enabled` gets a 403 on every call. ### CLI access The `argocd` CLI talks gRPC, which the browser route cannot carry: Cilium's Gateway translates gRPC-web into native gRPC, and native gRPC needs an HTTP/2 backend, which the plain `http` port is not. The chart therefore adds an `h2c` port (`http2`, 81) and a `GRPCRoute` on `argo-grpc.infra.k8s.wlkr.ch`, a hostname of its own so the browser route stays as it is. The Gateway offers no ALPN, and gRPC clients refuse TLS without it, so the CLI needs ALPN enforcement off: ```bash export GRPC_ENFORCE_ALPN_ENABLED=false argocd login argo-grpc.infra.k8s.wlkr.ch --sso --grpc-web ``` ## Health check ```bash kubectl -n argocd get applications ``` Every Application should be `Synced` and `Healthy`. One that stays `OutOfSync` has exhausted its retries and waits for a hand sync — see [Sync policy](). ## Pitfalls > [!WARNING] > > **The local admin is disabled; Authentik is the login** > > With Authentik down there is no way into the UI. The break-glass path is a pull request that re-enables `admin.enabled`, since selfHeal reverts a `kubectl patch`; the steps are in [When Authentik is down](). ## Recovery ### Reinstalling the chart ArgoCD manages ArgoCD, and the ApplicationSet that deploys the cluster sits inside that same Application. When a broken ArgoCD config has been synced by ArgoCD, `make install-argo` reinstalls the chart from the deployment host, and `kubectl apply -f payload/argocd/applicationset-platform.yaml` re-applies the rollout itself. # Authentik [Authentik]() is the cluster's identity provider. Every platform UI sits behind one login instead of a chart-default password each — and Hubble UI, which ships no authentication at all, would otherwise publish every network flow in the cluster to anyone who could reach the hostname. | Service | Authentication | | --- | --- | | ArgoCD | OIDC, local admin **disabled** | | Grafana | OIDC, login form disabled | | Home Assistant | Proxy outpost **in front of** its own login — see [Pitfalls](<#pitfalls>) | | Umami | Proxy outpost **in front of** its own login; the tracker paths pass through — see [Pitfalls](<#pitfalls>) | | Rook dashboard, Hubble UI, Prometheus, Alertmanager, Flowscape, Extractor admin UI | Proxy outpost | | Wollbi-Fescht, Wollbi Adventsfenster | Proxy outpost on the admin paths, with no `infra-users` binding, see [Groups and roles](<#groups-and-roles>) | ## At a glance | | | | --- | --- | | Namespace | `authentik` | | Depends on | [OpenBao]() for its OIDC client secrets, [CloudNativePG]() for the database, [Rook-Ceph]() under it | | If it is down | Every platform UI loses its only login — see [When Authentik is down](<#when-authentik-is-down>) | | Health check | `kubectl -n authentik get pods,clusters` → server and worker Ready, `authentik-db` healthy | | UI | `auth.k8s.wlkr.ch` | | Files | `payload/platform/authentik/` | ## Configuration ### Two integration styles **OIDC**, for applications that can do it themselves: ArgoCD and Grafana map group membership to a role. **Proxy**, for the rest: the hostname resolves to Authentik's embedded outpost, a reverse proxy that authenticates the request and only then forwards it to the backend. It needs nothing from the application and protects exactly as far as nothing else can reach the backend. ``` flowchart LR U([Browser]) --> GW[infra-gateway / apps-gateway] GW -->|argo, monitoring, OIDC workloads| APP[ArgoCD / Grafana / OIDC workloads] APP -.->|OIDC redirect to auth.k8s.wlkr.ch| AK[Authentik] GW -->|hubble, rook, prometheus, alertmanager, home, flowscape, analytics, extractor-admin| AK AK -->|authenticated| BE[Hubble UI / Ceph dashboard / ...] ``` ### One hostname Authentik answers on `auth.k8s.wlkr.ch` only, on `apps-gateway`, which the LAN reaches through the router's local record and the outside through Route53. Every OIDC client and the embedded outpost use that one name. The outpost is why there is only one. A proxied application's route points at `authentik-server`, but the login itself is an OAuth dance: the outpost sends the browser to the outpost's `authentik_host` to authenticate and gets it back on the application's own hostname. With Authentik on two names and `authentik_host` on the LAN-only one, an application on `apps-gateway` worked at home and redirected outside users to a name that does not resolve. `blueprints.yaml` pins `authentik_host` because the embedded outpost otherwise fills it from `web.base_url` on first start and never updates it. > [!WARNING] > > **A client must never mix names** > > The `iss` claim is the hostname the token was issued through. A discovery URI on one name and a redirect on another fails every login with an error that blames the token. Should a second hostname ever come back, pin each client to exactly one. ### Where the proxied routes point A proxied application's route lives with the application: Hubble's in `payload/platform/cilium/`, Rook's in `payload/platform/rook-ceph/`, Prometheus's and Alertmanager's in `payload/platform/monitoring/`, and Home Assistant's, Flowscape's, Umami's, Wollbi-Fescht's, Wollbi Adventsfenster's and the Extractor admin UI's in the [workloads repository](), each with `authentik-server` as `backendRef`. Gateway API forbids a cross-namespace `backendRef` unless the target namespace grants it, so `referencegrant.yaml` here allows `HTTPRoute` objects from exactly those namespaces to that Service only. A proxied workload is therefore two pull requests, by design: a workload cannot take itself out from behind the authentication layer on its own. ### Database `database.yaml` is a CloudNativePG `Cluster` with replicas, the only one on the cluster that has any. Every login on every platform UI goes through Authentik, so its database must survive a node drain: with one instance a reboot kills the database; with replicas, one takes over within a minute, once `enablePDB: false` lets the drain evict the primary. The chart's own Postgres subchart is off — both rules are in [the contract](). The replicas only survive a node loss while they sit on separate nodes, which the operator's own `preferred` anti-affinity does not guarantee — hence the `topologySpreadConstraints` block, under the rule in [Replica placement](). ### Chart values | Setting | Why | | --- | --- | | `authentik.disable_update_check`, `authentik.disable_startup_analytics` and `authentik.avatars` | Telemetry is switched off in config, not permitted by network policy — the same rule as Grafana, Loki and Alloy. `avatars` only seeds a fresh database: since 2024.2 the setting lives in **System → Settings**, and an existing install keeps calling Gravatar until it is set to `initials` there. Blueprints cannot set it: system settings are an internally managed model, and upstream declined a config-level override in [goauthentik/authentik\#26470]() | | `authentik.web.base_url` | Authentik builds e-mail links and outpost redirects from it and cannot infer it; unset, every admin page shows "The base URL has not been configured". The chart value backfills the tenant, so a rebuilt cluster needs no click | | `metrics.enabled` and `metrics.serviceMonitor.enabled` | The ServiceMonitor renders only when both are set; the switch alone produces nothing, silently. The worker is scraped too: tasks, outpost state and blueprint runs are measured there | | `postgresql.enabled: false`, `authentik.postgresql.host: authentik-db-rw` and the `global.env` entry | The database is the `Cluster` above. The password is the one CloudNativePG generated into `authentik-db-app`, read as `AUTHENTIK_POSTGRESQL__PASSWORD` from that Secret; it is never copied into OpenBao or Git | | `server.topologySpreadConstraints` | The chart's `podAntiAffinity` preset is `soft`, which is not enough to hold the replicas apart — see [Replica placement](). The worker is not replicated and needs none | | `worker.podAnnotations` `homelab.wlkr.ch/secret-generation` | Bumped whenever `authentik-secrets` or a workload's `authentik-secrets-` gains a key, so ArgoCD restarts the worker in the same sync — see [Pitfalls](<#pitfalls>) | | Workload blueprints as a `projected` volume, `optional: true` | `blueprints.configMaps` renders a plain `configMap` volume, and the kubelet refuses a pod whose ConfigMap is missing. Workload blueprints arrive with the [workloads](), which a fresh cluster may not have yet, so a required mount would keep the worker from starting | ### Where a blueprint lives Providers and applications are blueprints, so a rebuild reproduces them. `!Find` resolves objects Authentik ships, `!KeyOf` references another entry in the same blueprint, and `!Env` reads an environment variable, which keeps client secrets out of Git. `blueprints_discovery` runs on worker startup, hourly and on a file watcher, asynchronously, so anything configuring itself against a provider must tolerate it not existing yet. | What | Lives in | Why there | | --- | --- | --- | | Platform providers, applications, the outpost | `blueprints.yaml` | Mounted into the worker as a ConfigMap | | A workload's provider and application | The workload's directory in the workloads repository, as the ConfigMap `authentik-blueprint-` targeted at `authentik` | The provider and the application it authenticates change in one commit; the [`authentik-foreign-objects`]() policy refuses any other object a workload sends here | | Client credentials | Generated by a `PushSecret` beside their reader — `secrets.yaml` for the platform's, the workload's own `secrets.yaml` for a workload's — and read here by `secrets-.yaml` | A workload can mint a pair but not register it: the `ExternalSecret` and mount entry here keep SSO onboarding a deliberate two-repository act | | The mount entry and `secret-generation` | `application.yaml` | One projected-volume source per workload, with or without credentials; a new credential needs the worker restarted | | The outpost's `providers:` list | `blueprints.yaml` | It replaces one global object; two repositories writing it would overwrite each other | | `referencegrant.yaml` | Here | A proxied workload's route needs granting from this side | ### One issuer for OpenCloud OpenCloud's web, desktop, iOS and Android clients are four OIDC clients with IDs fixed by the clients, so four providers in Authentik, each under an application of its own. The server and the native clients verify every token against one issuer, and Authentik derives a provider's issuer from its application slug, so the four would disagree. All four therefore use `issuer_mode: global`, whose issuer is the bare hostname, and the `HTTPRoute` rewrites `/.well-known/openid-configuration` on that hostname to the `opencloud` application's discovery document, which is where OpenCloud and its clients fetch it. The document names the global issuer and the shared endpoints, so it serves the other three providers as well. Nothing else uses the root path: every other provider is found under its slug. ### User settings flow Dependency-Track keys its accounts on the `preferred_username` claim, so a user who could rename themselves could take over another's account. `05-user-settings.yaml` therefore gives the default brand its own user settings flow with no username field; a username changes only under **Directory → Users**. It replaces the brand's flow instead of editing the default one because the default blueprint rewrites its own objects on every upgrade, and because the **System → Settings** switch `default_user_change_username` is not a model a blueprint can set. ### Authentication flow `auth.k8s.wlkr.ch` is reachable from the Internet, and the default authentication flow lets a user without a device in on a password alone. `06-authentication.yaml` gives the default brand its own flow out of the default identification, password and login stages, with two additions. It is an own flow for the same reason as the [user settings flow](<#user-settings-flow>): the default blueprint rewrites its own objects on every upgrade. | Addition | Why | | --- | --- | | Authenticator validation with `not_configured_action: configure`; TOTP and WebAuthn as setup stages, static tokens accepted as a device class | A password alone is one leak away from every application, so a user without a device enrols one at the next login instead of being let through. Static tokens are the recovery codes for a lost phone, enrolled afterwards under **Settings → MFA devices**, which is why they are not offered as the first device | | A deny stage after identification, selected by a reputation policy on the address and the username with threshold `-5` | Every failed login scores both `-1`, so after five an address or a username is refused for a day (`AUTHENTIK_REPUTATION__EXPIRY`); the password stage's own five-attempt cancel only ends the current attempt. The policy *passes* for a bad score, so it selects a deny stage rather than guarding the flow, and it is evaluated when the stage is reached because the flow's own policies see an anonymous request before the username is known | To lift a lockout early, delete the score under **Customization → Policies → Reputation scores**. ### Bootstrap credentials Authentik's built-in bootstrap blueprint creates `akadmin` from `AUTHENTIK_BOOTSTRAP_PASSWORD` only while no `akadmin` exists, and reads nothing from it afterwards. `06-authentication.yaml` then sets the account `is_active: false`: a password-only superuser that never logs in is the one worth guessing, and a disabled one authenticates nowhere. The blueprint touches only `is_active`, so the password and the `authentik Admins` membership stay for break-glass. Break-glass is `ak shell` from a cluster-admin kubeconfig, which nothing in Authentik can lock out: ```bash kubectl -n authentik exec deploy/authentik-server -c server -- ak shell -c \ 'from authentik.core.models import User; User.objects.filter(username="akadmin").update(is_active=True)' ``` Then log in with `bootstrap-password` from OpenBao — see [Generated secrets]() — enrol a second factor like any user, and switch the account off again with the same command and `is_active=False`: a blueprint is re-applied only when its file changes, so nothing switches it off for you. | Variable | Set | Why | | --- | --- | --- | | `AUTHENTIK_BOOTSTRAP_PASSWORD` | Always, from `bootstrap-password` | Without it a fresh install creates `akadmin` with no usable password, and the initial-setup flow that sets one is open to whoever reaches `auth.k8s.wlkr.ch` first. Blanking it after the first start buys nothing: the account keeps the password | | `AUTHENTIK_BOOTSTRAP_TOKEN` | Never | It mints a non-expiring superuser API token, `authentik-bootstrap-token`, which nothing here uses, and while the variable is set the blueprint recreates the token whenever it is deleted | A cluster installed before the token was dropped still holds it: delete `authentik-bootstrap-token` under **Directory → Tokens and App passwords**, once the worker has restarted without the variable. ### Groups and roles Authorisation is group membership. The blueprints create these groups; membership is set by hand under **Directory → Groups**: | Group | Grants | | --- | --- | | `infra-users` | Past the Authentik login of every infra application: ArgoCD, Grafana, Hubble, the Ceph dashboard, Prometheus, Alertmanager, Flowscape and the Extractor admin UI. Grafana `Viewer` for a direct member | | `argocd-admins` | ArgoCD `role:admin`, and `infra-users` | | `argocd-viewers` | ArgoCD `role:readonly`, and `infra-users` | | `grafana-admins` | Grafana `Admin`, and `infra-users` | | `grafana-editors` | Grafana `Editor`, and `infra-users` | | `wollbi-editors` | A Payload editor on advent.wollbi.ch and fest.wollbi.ch, created on first sign-in; declared in the two sites' blueprints in the workloads repository | Every infra application carries a policy binding to `infra-users`, so an account outside it, such as a wollbi editor's, is refused by Authentik before the application sees it. The four role groups have `infra-users` as their parent, and Authentik counts a child group's members as members of the parent. Grafana only sees the groups a user belongs to directly, and with `role_attribute_strict` it refuses a login that matches none of its three. ArgoCD's `policy.default` is empty, so a user in neither ArgoCD group gets **no** access. The wollbi sites bind `wollbi-editors` instead. Home Assistant and Umami have no binding: they have logins of their own. ## Usage ### Client secrets The OIDC client IDs and secrets are generated once into OpenBao and read by both sides — Authentik through `!Env`, ArgoCD and Grafana through their own `ExternalSecret` — so a rebuilt Authentik gets the same credentials and nothing is copied out of a UI. The `PushSecret` in `secrets.yaml` writes `kv/authentik/config` once — see [Generated secrets]() — with the secret key, the bootstrap password and the ArgoCD and Grafana client pairs; its `Password` generator's `secretKeys` is the list of keys. A workload's client credentials go in the workload's own path (Dependency-Track's in `kv/dependency-track/config`, each read by its own `secrets-.yaml`, mounted `optional`): rewriting `kv/authentik/config` would rotate the secret key that signs every session and token, and External Secrets fails an `ExternalSecret` whole when one key is missing, so a shared one would leave a cluster without the workload with no `AUTHENTIK_SECRET_KEY` either. The first administrator is created through `akadmin`, switched on for the occasion as in [Bootstrap credentials](<#bootstrap-credentials>); the steps are in [Quickstart step 12](). ### Adding an application 1. Generate its client credentials into OpenBao with a `PushSecret` beside the workload's own `ExternalSecret`, and read them here with an `ExternalSecret` mounted `optional`, like `secrets-dependency-track.yaml`. A public client (PKCE, as OpenCloud's web and native clients) has no credentials: its client IDs are fixed by the client and sit in the blueprint in clear, so this step and step 3 do not apply. 2. Mount the workload's blueprint: in `application.yaml`, a `configMap` source named `authentik-blueprint-` with `optional: true` under `worker.volumes`, in the `projected.sources` of `blueprints-workloads`. 3. Hand the worker the credentials: in the same file, a `secretRef` to `authentik-secrets-` with `optional: true` under `global.envFrom`, and bump `homelab.wlkr.ch/secret-generation`. 4. Add the provider and application blueprint: an OAuth2 provider with `grant_types`, or a proxy provider added to the outpost's `providers:` list. 5. For a proxied application, point its `HTTPRoute` at `authentik-server`, add its namespace to `referencegrant.yaml`, and add a `toEndpoints` rule for its pods to the `authentik-server` policy in `payload/platform/security/network-policies/authentik.yaml`: the outpost dials the backend, and without the rule the login succeeds and the request after it times out. 6. Bind `infra-users` to an infra application in the blueprint, as an `authentik_policies.policybinding` with the application as `target`. An application with no binding is open to every Authentik account. ### Adding a user 1. Create the account under **Directory → Users**. 2. Add it to `infra-users`, or to a role group from [Groups and roles](<#groups-and-roles>), under **Directory → Groups**. Membership is not blueprinted because that would put a username in the repository, and only a superuser passes the bindings without it. ## Health check When every login fails, in this order: 1. Server and worker are Ready and the database is healthy: ```bash kubectl -n authentik get pods,clusters ``` 2. The discovery document names the one hostname as `issuer`: ```bash curl -s https://auth.k8s.wlkr.ch/application/o/argocd/.well-known/openid-configuration | jq .issuer ``` 3. The embedded outpost is healthy (`Outposts` in the admin UI, or): ```bash kubectl -n authentik logs deploy/authentik-server | grep -i outpost | tail ``` 4. The worker applied the blueprints — a failed one names its file: ```bash kubectl -n authentik logs deploy/authentik-worker | grep -i blueprint | tail ``` ## Pitfalls > [!CAUTION] > > **A new client credential needs the worker restarted** > > `envFrom` injects `authentik-secrets` **once, when the pod starts**. A blueprint reading a newly added credential through `!Env` on a running worker gets an empty string and creates a provider that answers 404 on its discovery endpoint. Bumping `secret-generation` restarts the worker in the sync that adds the credential; `kubectl rollout restart` fixes a running cluster but leaves nothing for the next one. > [!CAUTION] > > **A workload blueprint is code with admin rights** > > The worker applies a blueprint with full access to Authentik's models, so a blueprint ConfigMap from the workloads repository can create a superuser or add someone to `argocd-admins`. The admission policy limits which ConfigMap a workload may write, not what it says: review a blueprint change in the workloads repository like a change here. > [!WARNING] > > **OAuth2 providers must name their grant types** > > `grant_types` defaults to an empty list. A provider that omits it serves no grants, and every login fails with `invalid_request: The request is otherwise malformed`. > [!WARNING] > > **The outpost entry replaces its provider list** > > `authentik_outposts.outpost` sets `providers` wholesale. Every proxied application must be listed there; adding one and forgetting the list silently unassigns the others, which fail open. > [!NOTE] > > **Home Assistant and Umami are the exceptions** > > Each has a login of its own and upstream ships no OIDC provider to replace it, so the outpost sits *in front of* it: browser users authenticate twice, which is defence in depth rather than single sign-on. Home Assistant's companion apps and webhooks hold a long-lived token and cannot complete an interactive login, so `skip_path_regex` lets `/api/`, `/auth/token` and the external-auth callback through; Home Assistant's own accounts are the only thing guarding those paths, and revoking an Authentik account does not revoke a Home Assistant token. Umami's tracker script and collect endpoint are loaded by every visitor of every site that embeds it, so `/script.js` and `/api/send` pass the same way; the rest of the dashboard and API stays behind the login. ## Recovery ### When Authentik is down Single sign-on is a single point of failure for logging in at all. The ArgoCD path needs a working `kubectl`, so keep a kubeconfig off-cluster. **ArgoCD** — re-enable the local admin account by merging a pull request that sets `admin.enabled` in `payload/argocd/values.yaml`; the `argocd` Application self-heals, so a `kubectl patch` on `argocd-cm` is reverted at the next reconcile. ArgoCD syncs the merge without a login. Then restart the server, which generates `argocd-initial-admin-secret`, and read the password as in [Quickstart](). Revert the pull request once Authentik is back. ```bash kubectl -n argocd rollout restart deploy/argocd-server ``` **Grafana** — the login form is hidden, not removed: the admin account works through the API, and `GF_AUTH_DISABLE_LOGIN_FORM=false` brings the form back. **Proxied dashboards** have no bypass; port-forward instead: ```bash kubectl -n kube-system port-forward svc/hubble-ui 8080:80 kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 ``` ### ArgoCD's API key account Disabling the local admin also removes its `apiKey` capability. If a token is needed, add an account scoped to what the token is for rather than re-enabling admin: The `claude` account is one, for the in-cluster agents; see [Agent account](). The shape: ```yaml configs: cm: accounts.mcp: apiKey rbac: policy.csv: | p, role:mcp, applications, get, */*, allow g, mcp, role:mcp ``` # cert-manager TLS certificate automation via Let's Encrypt, using DNS-01 challenges through AWS Route53. DNS-01 because these hostnames resolve to RFC1918 addresses that Let's Encrypt cannot reach, and because it is the only way to get a wildcard: one certificate per zone, listed in `payload/platform/certificates/certificates.yaml`, covers every hostname the Gateways terminate. ## At a glance | | | | --- | --- | | Namespace | `cert-manager`; the wildcards land in `kube-system` beside the Gateways, the agent sessions' certificates in their own namespaces; see [SSH listener]() | | Depends on | [External Secrets]() for the Route53 credential, so transitively on [OpenBao]() | | If it is down | Nothing immediately. Certificates stop renewing, and the consequence surfaces up to sixty days later | | Health check | `kubectl get certificate -A` → all `READY=True` | | Files | `payload/platform/cert-manager/`, `payload/platform/certificates/` | ## Configuration | Setting | Why | | --- | --- | | Issuers and certificates in their own `certificates` Application | Kept with cert-manager, the controller would read Degraded until OpenBao held the Route53 credentials — see [Bootstrap convergence]() | | Three sync waves inside `certificates`: `ExternalSecret`, then the ClusterIssuers, then the Certificates | In one wave ArgoCD orders custom resources alphabetically — `Certificate`, `ClusterIssuer`, `ExternalSecret`, exactly backwards — and an issuer applied before its Secret stays `Ready=False` with `InvalidSolver` until something resyncs it | | Sync `retry` on both Applications | Without one a failed apply ends the operation where it fell; one flake at the front of the chain, such as the ESO webhook being unreachable on a fresh CNI, leaves every issuer and certificate behind it unmade | | `ServiceMonitor` rendered unconditionally | The chart cannot sync until the Prometheus operator CRDs exist, which is why they are the separate `prometheus-operator-crds` Application — see [Monitoring]() | | Two webhook replicas, a PodDisruptionBudget and a hostname spread | The webhook's `failurePolicy` is `Fail`, so while a single replica reschedules through a Kured drain every Certificate and Issuer apply is refused — see [Replica placement]() | | `letsencrypt-staging` and `letsencrypt-prod` | Use staging first: production allows five duplicate certificates per week, a misconfigured solver retries until that is gone, and there is no appeals process | The IAM user needs at minimum: ```json { "Effect": "Allow", "Action": ["route53:GetChange", "route53:ChangeResourceRecordSets", "route53:ListHostedZonesByName"], "Resource": "*" } ``` ## Usage Store the credentials once OpenBao and ESO are up; `make bao-secrets` prompts for them, or by hand: ```bash bao kv put kv/cert-manager/route53 \ access-key-id="YOUR_AWS_ACCESS_KEY_ID" \ secret-access-key="YOUR_AWS_SECRET_ACCESS_KEY" ``` ESO creates the `route53-credentials` Secret within its `refreshInterval`, or [immediately on request](). ## Health check Work down the chain of custody; the answer is nearly always further back than the `Certificate`: ```bash kubectl describe certificate -n kube-system kubectl get certificaterequest,order,challenge -A kubectl -n cert-manager logs deploy/cert-manager --tail=100 ``` A `Challenge` stuck in `pending` is a DNS problem: either the credentials cannot write to the zone, or the TXT record is there and the resolver has not caught up. `dig +short TXT _acme-challenge.` settles which. ## Pitfalls > [!NOTE] > > **A sealed OpenBao means no renewals** > > Until OpenBao is unsealed and the secret stored, cert-manager cannot issue or renew, and the failure surfaces as an expired certificate roughly two months later — see [OpenBao](). # Cilium Cilium is the CNI, and on this cluster considerably more: it replaces `kube-proxy`, terminates ingress through Gateway API, hands out LoadBalancer addresses on a network with no cloud load balancer, encrypts node-to-node traffic, and shows what is talking to what. That consolidation is the argument for it on bare metal, and the reason a broken Cilium is never a small problem. ## At a glance | | | | --- | --- | | Namespace | `kube-system` | | Depends on | The Gateway API and Prometheus operator CRDs it renders against, and a `k8sServiceHost` that answers | | If it is down | Everything. No CNI, no service routing, no ingress, no LoadBalancer addresses | | Health check | `kubectl -n kube-system exec ds/cilium -- cilium status --brief` | | UI | `hubble.infra.k8s.wlkr.ch` (Hubble); the same flows in 3D across namespaces at [Flowscape]() | | Files | `payload/platform/cilium/` | ## Configuration | Setting | Why | | --- | --- | | `kubeProxyReplacement: true` | Service routing in eBPF. `proxy.disabled` in `ansible/templates/kubeadm.yaml.j2` keeps `kube-proxy` out at init and on every `kubeadm upgrade apply` | | `k8sServiceHost` as a literal IP | With `kube-proxy` gone Cilium cannot reach the API server through a Service, so it names the [control-plane VIP]() — see [Pitfalls](<#pitfalls>) | | Gateway API | Replaces an Ingress controller — see [Gateway API]() | | `lb-pools.yaml` | One address each for the apps and infra Gateways, announced over L2 ARP so the LAN can find them | | `bgp.yaml` | The same two addresses as `/32` routes over eBGP from every control-plane node to the router. The router forwards traffic it routes, from other VLANs, VPN clients and port forwards, to all three nodes at once and drops a dead one after the hold time, where L2 moves one lease. L2 stays because the addresses sit inside the flat LAN, whose hosts ARP for them and never ask the router. The ASNs and timers are in `bgp.yaml`. The router side is FRR on the UDM Pro: `make config` renders it from the inventory's `router_*` variables into `output/router/frr-bgp.conf`, so a firmware reset is recovered by pasting that file | | `encryption.type: wireguard` | Transparent node-to-node encryption | | Hubble, behind the [Authentik]() proxy outpost | Hubble has no authentication of its own; the HTTPRoute points at the outpost and a ReferenceGrant in `payload/platform/authentik/` permits the cross-namespace reference | | Hubble metrics labelled by namespace | The chart's Hubble dashboards filter on source and destination namespace and show nothing without them; namespace keeps the series count to the square of the namespace count. The HTTP metric adds workloads, but only for traffic Cilium proxies at L7, which is the Gateway. | | `hubble.export.static` | Every flow with a `DROPPED`, `ERROR` or `AUDIT` verdict, and every HTTP request a policy proxies at L7, goes to a file on the node that [Alloy]() tails into Loki, with a field mask that keeps identities, ports and the request line and drops the rest. Hubble's own buffer is minutes deep; this is the record of what a policy refused after the fact, and of the requests the L7 rules are tightened from. Other forwarded flows are not exported: thousands a second on a cluster this size, where the proxied requests are a few. | | `hubble.redact` | Every header except `User-Agent` and `Content-Type`, and the password of a basic-auth URL, is replaced before a flow leaves the agent, for the export file and Relay alike. An L7 rule proxies the requests that carry credentials: External Secrets' `X-Vault-Token`, the bearer tokens Prometheus scrapes with, Velero's signed S3 requests, Authentik's cookies. The method, path and query the L7 rules are tightened from survive | | `hostFirewall.enabled: true` | Filters traffic to the node's own addresses, which is every host-networked port, with `CiliumClusterwideNetworkPolicy` objects that select nodes. Attaching it denies nothing on IPv4 until a policy selects the node; IPv6 is another matter, see [Pitfalls](<#pitfalls>). Rollout and policy in [Host firewall]() | | `policyAuditMode: false` | Policies are enforced. On, verdicts are logged and not enforced, which is how a rollout's observation week runs — see [Security Policies](). Cluster-wide, and blind to L7 rules | | `rollOutCiliumPods` and the four `rollOutPods` | The agent, Envoy, the operator, Relay and the UI read their ConfigMap once, at startup. With a checksum of it on the pod template, a merged value rolls the pods and is live within minutes, and a value the agent refuses fails at merge time, while someone is watching, not at the next unrelated restart. The price is that every change to `cilium-config` is a rolling restart of the CNI on every node: running pods keep their networking, policy updates pause per node for the seconds its agent is down, and the [`k8sServiceHost` pitfall](<#pitfalls>) bites at merge, not later | | `updateStrategy.rollingUpdate.maxUnavailable: 1` | The chart default restarts the agent on two nodes at once, which can be two of the three control planes that hold the L2 leases and BGP sessions; one at a time keeps every Gateway address announced throughout a rollout | | `operator.replicas` above one | The operator owns Gateway API reconciliation, LoadBalancer IP allocation and the `cilium-secrets` sync; with one replica, a drain or crash of its node stalls all three until the reschedule. The chart's hostname anti-affinity keeps the replicas apart, so the standby is already running — see the [operator pitfall](<#pitfalls>) | | `lb-pools.yaml` on `cilium.io/v2` | The storage version for `CiliumLoadBalancerIPPool`; `v2alpha1` is served as deprecated and warns on every sync. `CiliumL2AnnouncementPolicy` has no `v2` yet and stays | | `cilium-agent` memory limit | It is the one process whose death takes pod networking with it, so it was sized from a week of measurement, not a day, before the [platform rule]() applied | | `trustCRDsExist: true` | The chart otherwise refuses to render while `monitoring.coreos.com/v1` is missing: the bootstrap install, a render before `prometheus-operator-crds` has synced, and the diff preview's throwaway cluster | ## Usage ### Installation Cilium is installed twice, in a sense. `make install-cilium` installs it by Helm, because no pod runs without a CNI and ArgoCD is a pod; ArgoCD then adopts the release. The version comes from `targetRevision` in `application.yaml` — see [Version pins](). The bootstrap install uses the same `values.yaml`, and has to: the pods roll whenever `cilium-config` changes, so a slimmer bootstrap config would restart every agent the moment ArgoCD adopts the release. The Gateway API CRDs go in first because the operator checks for them once, at startup. The one difference is the three `serviceMonitor.enabled` flags, which `make` turns off because the Prometheus operator CRDs arrive through ArgoCD; ArgoCD adds the monitors back on adoption, rolling the agent and operator once. ## Health check ```bash kubectl -n kube-system exec ds/cilium -- cilium status --brief ``` The BGP session on each control-plane node, `established` with two routes advertised. `bgp.yaml` selects the control planes only, so ask that node's agent rather than whichever pod `ds/cilium` resolves to: ```bash kubectl -n kube-system exec "$(kubectl -n kube-system get pod -l k8s-app=cilium \ --field-selector spec.nodeName= -o name)" -- cilium bgp peers ``` Hubble answers *what did I just break?* during a network policy rollout — see [Security Policies](). The `hubble` CLI talks to Relay through a port-forward on its default port: ```bash kubectl -n kube-system port-forward deploy/hubble-relay 4245:4245 & hubble observe --verdict DROPPED --follow ``` The same verdicts, for the whole Loki retention window, in Grafana: ```text {job="hubble", verdict="DROPPED"} | json | flow_drop_reason_desc="POLICY_DENIED" ``` ## Pitfalls > [!CAUTION] > > **`k8sServiceHost` pointed at an address that does not answer takes the cluster down** > > Every Cilium pod loses the API server at once, and cluster networking goes with it — including whatever you were using to fix the problem. The change ordering is in [Control Plane VIP](). > [!WARNING] > > **An operator that starts without the API server leaves the Gateways without certificates** > > cilium-operator checks for the Gateway API CRDs once at startup; if that check fails it runs without its Gateway controller until the next restart, and `cilium-secrets` empties. Envoy then resets the TLS handshake on every node without a cached copy, so some clients fail and others do not, depending on which node the router hashes them to. `CiliumGatewayCertificatesMissing` fires; `kubectl -n kube-system rollout restart deploy/cilium-operator` recovers it. > [!CAUTION] > > **A chart default that disappears stays in `cilium-config`** > > The bootstrap install's `helm` field manager still co-owns every `cilium-config` key it wrote, so ArgoCD's server-side apply cannot delete one the chart stops rendering. When a chart release dropped the `envoy.xdsMode` default, the agents kept `envoy-xds-mode: ads` while Envoy's bootstrap switched to split xDS; Envoy got no listeners, and every Gateway and L7 policy stopped forwarding. Pin such a value in `values.yaml` rather than relying on the default. > [!WARNING] > > **The host firewall drops all node IPv6, with or without a policy** > > With `enable-ipv6: false` the datapath has no IPv6 branch, so on the native device and `cilium_host` every IPv6 packet is dropped as `DROP_UNKNOWN_L3` in both directions, before any policy or audit mode ([upstream]()). A node keeps its SLAAC address until the lifetime runs out, and a client that does not fall back to IPv4 stalls until then. The drops reach Loki as `DROPPED` flows with a reason other than `POLICY_DENIED`, so `HubblePolicyDrops` stays quiet. # CloudNativePG Every chart that needs PostgreSQL ships its own, each upgraded by whoever maintains the chart that buried it. CloudNativePG is one operator in `cnpg-system` and a `Cluster` resource per database. ## At a glance | | | | --- | --- | | Namespace | `cnpg-system` for the operator; each database in its workload's namespace | | Depends on | [Rook-Ceph]() for the database volumes and the backup buckets; [cert-manager]() for the backup plugin's gRPC certificates | | If it is down | Running databases keep running; failover, upgrades and new clusters stop | | Health check | `kubectl get clusters.postgresql.cnpg.io -A` → `Cluster in healthy state` | | Backups | Continuous WAL and a nightly base backup per database, to a bucket of its own | | Files | `payload/platform/cloudnative-pg/` | ## Configuration A `Cluster` is not a StatefulSet with a nicer name: | It handles | Instead of | | --- | --- | | Primary election and failover between instances | A single pod that takes the database down with it | | `-rw` and `-ro` Services that follow the primary | Hardcoding a pod name and being wrong after a restart | | Rolling minor-version upgrades, replicas first | A `StatefulSet` image bump and hope | | Generated credentials in a `-app` Secret | A password in Git, or a manual `kubectl create secret` | | A PodMonitor and an upstream Grafana dashboard | No visibility until something is already wrong | Major-version upgrades are declarative too, but offline: ArgoCD reports Degraded while one runs and the workload is down for its duration. Read [the upstream guide]() before merging one. Databases claim `rook-ceph-block`, so Ceph's three copies already sit under every database and a single-instance `Cluster` is a defensible default: the data survives a node loss even when the process does not. Where a workload cannot tolerate the restart, `instances: 3` is the change, as [Authentik]() shows. The operator itself runs two replicas (`replicaCount` in `application.yaml`'s `valuesObject`), spread per [Replica placement](): the same pod serves the `Cluster` and `ScheduledBackup` admission webhooks with `failurePolicy: Fail`, so a single replica rescheduling through a Kured drain refuses every apply, while leader election keeps one of the two reconciling. The chart ships no PodDisruptionBudget. ### Backups The [Barman Cloud plugin]() runs beside the operator and injects a sidecar into every instance that names an `ObjectStore`. The sidecar archives each WAL segment as PostgreSQL closes it and takes the base backups a `ScheduledBackup` asks for, so a database restores to any second inside the retention window, not just to last night. | Choice | Why | | --- | --- | | The plugin, not `spec.backup.barmanObjectStore` | The in-tree field is deprecated upstream and is removed in a later release | | One bucket per database, claimed in its own namespace | The bucket's keys exist only there, so a compromised workload can read or delete its own backups and nobody else's | | `https://s3.infra.k8s.wlkr.ch`, not the plain RGW Service | Backups hold every row of the database; the Gateway's TLS is the same path Velero uses | | Base backup nightly, after Velero | The two never contend for the object store; WAL archiving covers the hours between | | `retentionPolicy` equal to `schedules.ttl` in `payload/platform/velero/application.yaml` | One window to reason about: what Velero can no longer restore, the WAL archive cannot either. Change the two together: each `backup.yaml` restates the value | | Preview databases have none | They are rebuilt from the pull request and deleted with it | `ObjectStore` is a namespaced resource with no cluster-scoped counterpart, so the per-database part is a short, identical stanza (below); everything else is shared: the plugin, and the `cnpg-backups` policy in `payload/platform/security/network-policies/cluster-wide.yaml` that lets any CloudNativePG pod reach the object store. ## Usage ### The contract **Every workload that needs PostgreSQL gets a CloudNativePG `Cluster`. No chart-bundled database, ever.** A bundled Postgres moves when the application's chart decides — a major version jump inside someone else's patch release; a `Cluster` moves when its `imageName` changes, a Renovate PR with a human in front of it. Most bundled subcharts are also Bitnami's, whose registry terms keep changing. The rule for the [workloads repository](): 1. Disable the subchart (`postgresql.enabled: false` or `internalDatabase.enabled: false`) and point the chart at the `Cluster`. 2. Never copy the password. CloudNativePG writes `username`, `password`, `host`, `port`, `dbname` and a ready-made `uri` to `-app`; charts take those through an `existingSecret` block, and applications without one read `uri` as an environment variable. Nothing goes into Git or OpenBao — nothing outside the cluster ever needs this secret. 3. Put the `Cluster` in the workload's namespace, not `cnpg-system`, so the namespace's `CiliumNetworkPolicy` governs port 5432 and deleting the namespace takes the database with it. 4. Set `enablePDB: false`. The operator's `-primary` PodDisruptionBudget refuses the primary's eviction outright, so a node drain hangs on it until [Kured]() gives up and the node is never rebooted. Without the budget a single instance is down for the move to another node and a three-instance cluster fails over within a minute. 5. Back it up with the [stanza below](<#backing-up-a-database>). Velero's nightly PVC snapshot is crash-consistent and restores only to the night it was taken; the WAL archive restores to any second. ### Backing up a database Next to the `Cluster`, in the same namespace, with `` the cluster's name. The bucket name is cluster-wide, hence the namespace in it; the sync wave puts the store ahead of the `Cluster` that names it. ```yaml --- apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: -backup namespace: annotations: argocd.argoproj.io/sync-options: Prune=false argocd.argoproj.io/sync-wave: "-2" spec: bucketName: cnpg- storageClassName: ceph-bucket --- apiVersion: barmancloud.cnpg.io/v1 kind: ObjectStore metadata: name: namespace: annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true argocd.argoproj.io/sync-wave: "-2" spec: retentionPolicy: 14d configuration: destinationPath: s3://cnpg-/ endpointURL: https://s3.infra.k8s.wlkr.ch s3Credentials: accessKeyId: {name: -backup, key: AWS_ACCESS_KEY_ID} secretAccessKey: {name: -backup, key: AWS_SECRET_ACCESS_KEY} wal: compression: gzip data: compression: gzip --- apiVersion: postgresql.cnpg.io/v1 kind: ScheduledBackup metadata: name: namespace: spec: schedule: "0 0 3 * * *" immediate: true backupOwnerReference: self cluster: name: method: plugin pluginConfiguration: name: barman-cloud.cloudnative-pg.io ``` And in the `Cluster`: ```yaml plugins: - name: barman-cloud.cloudnative-pg.io enabled: true isWALArchiver: true parameters: barmanObjectName: ``` `enabled: true` is the webhook's default; written out, ArgoCD's diff matches the live object, since `plugins` is compared as a whole list. Adding the plugin restarts every instance to inject the sidecar: a single-instance database is down for that restart, a replicated one switches over. `immediate: true` takes the first base backup at once, so the WAL archive is restorable from the first day. ## Health check ```bash kubectl get clusters.postgresql.cnpg.io -A kubectl cnpg status -n ``` `Cluster in healthy state` is the phrase; ArgoCD's built-in health check reads the same field, which lets a sync wave wait for a database before the application in front of it. The plugin prints the primary, replication lag, WAL position and last failover. ```bash kubectl get backups.postgresql.cnpg.io -A # Phase completed, one per night kubectl get clusters.postgresql.cnpg.io -A \ -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?(@.type=="ContinuousArchiving")].status' kubectl get objectstores.barmancloud.cnpg.io -A -o yaml | grep -A3 serverRecoveryWindow ``` `ContinuousArchiving` `True` means the last WAL segment reached the bucket; `False` means WAL is piling up on the instance's volume, which fills it. The recovery window lives on the `ObjectStore`, per server name: the `Cluster`'s own backup fields and the `cnpg_collector_*_backup_timestamp` metrics stay empty with a plugin, which upstream documents as deprecated under CNPG-I. `prometheusrule.yaml` therefore reads the `ObjectStore` and `Backup` objects through kube-state-metrics' [custom resource metrics](): `CNPGBackupStale` when a server's last successful base backup is older than the rule's threshold, `CNPGBackupFailed` when a recent Backup ended in `failed`. Both are warnings on the daily mail. ## Pitfalls - **A stuck archive fills the volume.** PostgreSQL keeps every WAL segment the sidecar has not archived; a broken bucket or policy shows as `ContinuousArchiving` `False` long before the PVC is full. Fix the cause, never delete `pg_wal`. - **A dead replica fills the primary.** Its replication slot holds WAL until the replica catches up, so every multi-instance `Cluster` caps it with `max_slot_wal_keep_size` at a quarter of the volume; past that the slot is invalidated and the replica restores from the WAL archive or must be recreated with `kubectl cnpg destroy `. - **A full volume is not resized by the operator.** An instance stopped with `no free disk space for WALs` keeps its PVC size although `storage.size` grew; patch that PVC to the new size by hand, as [upstream documents]() under "Disk full failure". - **A restore is a new `Cluster`.** Recovery bootstraps a fresh cluster from the bucket; it never rewinds a running one. Give the new cluster its own `ObjectStore` destination, or its archive would mix with the one it was restored from. ## Recovery ### Point-in-time restore 1. Pick the target time (UTC) and check it lies inside the window: ```bash kubectl -n get objectstore -o jsonpath='{.status.serverRecoveryWindow}{"\n"}' ``` 1. In the workloads repository, add a second `Cluster` named `-restore` beside the original, which stays untouched: every Application prunes, so a renamed `Cluster` is a deleted one. It bootstraps from the old store, keeping the old name as `serverName`: ```yaml bootstrap: recovery: source: origin recoveryTarget: targetTime: "2026-10-21 12:00:00+00" externalClusters: - name: origin plugin: name: barman-cloud.cloudnative-pg.io parameters: barmanObjectName: serverName: ``` Its own `plugins` entry keeps the same `barmanObjectName` with `serverName: -restore`, so the new archive lands beside the old one. 1. Merge and wait for `Cluster in healthy state`; both databases now run, and the application still reads the original. 2. In a second PR, point the application at `-restore-app`, set the `ScheduledBackup`'s `cluster.name` to `-restore`, and delete the original `Cluster`; the sync prunes it. # Descheduler The scheduler places a pod once and never looks at it again. After [Kured]() drains a node, everything that lived there restarts on the others and stays there when the node comes back, so a cluster that has rebooted through an update ends up with one idle node and the rest loaded. The [descheduler]() runs on a schedule, evicts a few pods from the loaded nodes, and lets the scheduler place them where there is room. ## At a glance | | | | --- | --- | | Namespace | `descheduler` | | Depends on | [metrics-server]() for the node usage `LowNodeUtilization` compares | | If it is down | Nothing visible: pods stay where the last drain left them | | Health check | `kubectl -n descheduler get jobs` → the last run `Complete` | | Files | `payload/platform/descheduler/application.yaml` | ## Configuration The values are in `payload/platform/descheduler/application.yaml`; the reasoning: | Setting | Why | | --- | --- | | `kind: CronJob` with a `schedule` | A Deployment would poll continuously for a cluster that changes shape once per Flatcar update. A pass every few hours drifts a rebooted cluster back into balance the same day | | `nodeFit: true` | A pod is only evicted when another node has room for it, so an eviction never turns into a Pending pod | | `podProtections.extraEnabled: PodsWithPVC` | Ceph volumes would follow an evicted pod, but the pods that own them run alone: a moved [CloudNativePG]() primary is a failover and a cold restart of its application, a moved Loki a detach and reattach. Their PDBs are off so that Kured can drain, so this is the only guard. `emptyDir` pods stay protected at the default because their data goes with them | | System-critical pods at the default | `evictSystemCriticalPods` stays off, so nothing with `priorityClassName: system-cluster-critical` is ever moved. Those components hold their own placement rule instead — see [Replica placement]() | | `maxNoOfPodsToEvictPerNode` and `evictionLimits.node` | A few pods per node per pass: one Kured reboot already moved everything on a node once, a second mass move is what this component is meant to avoid | | `LowNodeUtilization` with `metricsUtilization.source: KubernetesMetrics` | Usage comes from metrics-server, not from requests. Requests here are still being tuned and many are far below what the pod uses, so a requests-based view would call a loaded node idle | | `thresholds` and `targetThresholds` | A node under the low mark is a candidate to receive, one over the high mark a candidate to lose pods; nodes in between are left alone | | `RemoveDuplicates` and `RemovePodsViolatingTopologySpreadConstraint` for both `DoNotSchedule` and `ScheduleAnyway` | After a drain, replicas of one workload sit on the same node; both plugins put them back on separate ones. Soft constraints count too, because the scheduler honoured them only until the drain | | `RemovePodsHavingTooManyRestarts` | A pod in a restart loop for hours is more likely stuck on its node than broken; a move is cheap and the restart count resets | | Metrics off | A CronJob pod exists for seconds; a `ServiceMonitor` would scrape an empty Service | ## Usage Nothing to operate. To watch a pass: ```bash kubectl -n descheduler create job --from=cronjob/descheduler manual-$(date +%s) kubectl -n descheduler logs -l app.kubernetes.io/name=descheduler --tail=100 ``` Every eviction is an event on the pod, so `kubectl get events -A --field-selector reason=Descheduled` shows what moved and why. ## Health check ```bash kubectl -n descheduler get cronjob,jobs kubectl -n descheduler logs -l app.kubernetes.io/name=descheduler --tail=100 | grep -i "evict\|error" kubectl top nodes # what LowNodeUtilization sees ``` ## Pitfalls - **It only evicts, never schedules.** Where a pod lands is the scheduler's decision; the descheduler assumes the scheduler will do better the second time, which holds after a drain and not much else. - **It honours PodDisruptionBudgets.** An eviction that would violate a PDB is refused. The single-instance [CloudNativePG]() clusters carry none, so that Kured can drain; `PodsWithPVC` is what keeps the descheduler off them. - **One pod per over-utilized node per pass** when usage comes from metrics-server. A badly skewed cluster takes several passes, which is the point of running it on a schedule rather than once. - **A move is a restart.** Anything without a second replica is briefly down when it is evicted; `nodeFit` guarantees a place to go, not zero downtime. # external-dns [external-dns]() creates the Route53 records for this cluster's hostnames from the `HTTPRoute`, `TLSRoute` and `GRPCRoute` objects that already declare them. Without it every hostname is a record made by hand in the AWS console, a step that lives nowhere in the repository and leaves a record pointing at nothing when the workload goes. ## At a glance | | | | --- | --- | | Namespace | `external-dns` | | Depends on | [External Secrets]() for its Route53 credential, [Gateway API]() for the HTTPRoutes it reads | | If it is down | New hostnames get no DNS record. Existing records are left alone | | Health check | `kubectl -n external-dns logs deploy/external-dns --tail=50` | | Files | `payload/platform/external-dns/` | ## Configuration In `application.yaml`'s `valuesObject`: | Setting | Why | | --- | --- | | Sources `gateway-httproute` and `gateway-grpcroute` | Every HTTP and gRPC hostname here is a route; the address comes from the route's parent `Gateway`, so nothing is written down twice. The `GRPCRoute` is Argo CD's, see [CLI access]() | | `target` annotation on `apps-gateway` | The cluster sits behind NAT, so the Gateway's own address is a LAN one nobody outside can reach. external-dns reads the public address off the Gateway, not the routes, so every `*.k8s.wlkr.ch` record gets it once. `infra-gateway` has none: those names stay LAN-only on purpose | | Source `gateway-tlsroute` | The agents' SSH hostnames are `TLSRoute`s, one per session. This version reads `TLSRoute` `v1`, the version installed, so each session's record appears with its route and no wildcard record exists to maintain; see [SSH listener]() | | Domain filter | Nothing outside that subtree is touched | | TXT registry with an owner id | A companion `_externaldns.*` TXT record stamps every record it creates, and it only modifies or deletes records carrying that stamp. Hand-made records in the same zone are invisible to it | | Policy `sync` | Deleting an HTTPRoute removes its record. Safe only because of the registry; without it `sync` would happily delete your MX records | | `--aws-zone-match-parent` | The records live in the `wlkr.ch` zone, not a zone of their own | | A second entry in `domainFilters` | `wollbi.ch` is a zone of its own in the same account, for the neighbourhood sites. Without it their routes are ignored rather than refused, which reads as external-dns being down | | `ssh.wlkr.ch` in `domainFilters` | Same zone as `k8s.wlkr.ch`, a different subtree | | Credentials as a file (`AWS_SHARED_CREDENTIALS_FILE`), not environment variables | The environment puts a key that can repoint every hostname into `kubectl describe pod`, crash dumps and every child process. The `ExternalSecret` templates an INI `credentials` key, the only key mounted; the original keys stay in the Secret so nothing still reading them breaks | The credential is a separate IAM user and OpenBao path from cert-manager's — see [OpenBao]() for why. Its policy needs `route53:ChangeResourceRecordSets` on the hosted zone, plus `route53:ListHostedZones` and `route53:ListResourceRecordSets`. ## Usage Nothing beyond the `HTTPRoute` you were already writing — see [Gateway API](). The credential is stored once, by `make bao-secrets` or by hand: ```bash bao kv put kv/external-dns/route53 \ access-key-id="AKIA..." \ secret-access-key="..." ``` ## Health check ```bash kubectl -n external-dns logs deploy/external-dns --tail=50 dig +short argo.infra.k8s.wlkr.ch kubectl -n external-dns port-forward deploy/external-dns 7979:7979 curl -s localhost:7979/metrics | grep endpoints_total # external_dns_source_endpoints_total 8 <- hostnames it can see # external_dns_registry_endpoints_total 0 <- records it owns ``` `All records are already up to date, there are no changes for the matching hosted zones` means it found no zone to change: sources fine, provider not, which on Route53 is usually zone matching. ## Pitfalls > [!NOTE] > > **A record that will not update is one external-dns does not own** > > Check for the matching `_externaldns.` TXT record in Route53. To adopt a hand-made record, create that TXT entry or delete the record and let external-dns recreate it. # External Secrets Operator [External Secrets Operator (ESO)]() bridges [OpenBao]() to native Kubernetes `Secret` objects. Workloads consume secrets the standard way — `envFrom`, `volumeMounts`, `imagePullSecrets` — with no sidecar, no init container and no code that knows what a Vault token is. ## At a glance | | | | --- | --- | | Namespace | `external-secrets` | | Depends on | [OpenBao]() at runtime, though not to be installed | | If it is down | Secrets already materialised keep working. Nothing rotates, and nothing new resolves | | Health check | `kubectl get clustersecretstore` → `Valid` on all three | | Files | `payload/platform/external-secrets/`; the stores in `payload/platform/openbao/cluster-secret-store.yaml` | ## Configuration ``` sequenceDiagram participant App as Workload participant ES as ExternalSecret participant ESO as ESO Controller participant Bao as OpenBao participant K as K8s Secret ESO->>Bao: login (k8s SA JWT) Bao-->>ESO: vault token ESO->>Bao: read kv/path Bao-->>ESO: secret data ESO->>K: create/update Secret App->>K: mount / envFrom ``` ESO mints a short-lived token for an `external-secrets-vault` ServiceAccount through the TokenRequest API, which the chart's controller ClusterRole allows in every namespace (`cluster-secret-store.yaml` binds its own `external-secrets-vault-token-creator` ClusterRole, scoped to that ServiceAccount name, beside it); OpenBao validates it against the TokenReview API and issues a token bound to that store's policy. Platform namespaces use the `openbao` store and the ServiceAccount in `external-secrets`; a workload uses `openbao-apps` and its own, which confines it to `kv//` — see [Kubernetes auth](). No long-lived credential exists anywhere. The stores use ESO's `vault` provider unchanged against the in-cluster Service. In `application.yaml`'s `valuesObject`: | Setting | Why | | --- | --- | | No `ClusterSecretStore` here | A store only validates against a running, unsealed OpenBao, so it ships with `openbao`, whose readiness it reports — see [Bootstrap convergence]() | | Webhook ServiceAccount token stays mounted | The webhook has no RoleBinding but builds an in-cluster client at startup and exits without the token (`unable to load in-cluster config`), which only shows when the pod is recreated | | Two webhook replicas, a PodDisruptionBudget and a hostname spread | The webhook's `failurePolicy` is `Fail`, so while a single replica reschedules through a Kured drain every `ExternalSecret`, `PushSecret` and store apply fails and the Argo sync that carried it goes to retry — see [Replica placement]() | ## Usage ### Adding a secret 1. Store the value in OpenBao, following the [KV layout](). A random value is not typed in at all: a `Password` generator and a `PushSecret` write it, see [Generated secrets](). ```bash bao kv put kv// key1="value1" key2="value2" ``` 2. Declare an `ExternalSecret` alongside the workload's other manifests; the parent Application picks it up. A workload names `openbao-apps` and ships the ServiceAccount that store logs in as; a platform component names `openbao` instead and needs no ServiceAccount: ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: external-secrets-vault namespace: my-app annotations: argocd.argoproj.io/sync-wave: "-2" automountServiceAccountToken: false --- apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: my-app-credentials namespace: my-app annotations: # ESO's CRDs arrive with the external-secrets Application; this # covers a sync before they have landed. argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: refreshInterval: 1h secretStoreRef: name: openbao-apps kind: ClusterSecretStore target: name: my-app-credentials # name of the resulting K8s Secret creationPolicy: Owner data: - secretKey: key1 # key in the K8s Secret remoteRef: key: my-app/ # the namespace is the prefix property: key1 # key inside the KV entry ``` To pull every key without listing them, use `dataFrom` instead of `data`: ```yaml spec: dataFrom: - extract: key: / ``` 3. Consume the Secret: ```yaml spec: containers: - name: my-app envFrom: - secretRef: name: my-app-credentials ``` `refreshInterval` polls OpenBao; `0` disables polling and re-syncs only on resource changes. To sync now rather than wait: ```bash kubectl annotate externalsecret -n \ force-sync=$(date +%s) --overwrite ``` ## Health check Start with the store; if it is not Ready, nothing downstream is either. `openbao-apps` and `openbao-claude-agents` report `Valid` without logging in, so their auth failures show on the `ExternalSecret` instead: ```bash kubectl get clustersecretstore openbao -o jsonpath='{.status}' | jq ``` | Cause | Fix | | --- | --- | | OpenBao is sealed | [Unseal it]() | | Kubernetes auth role missing | `make bao-policies` writes all three roles; it needs a [root token]() | | Wrong `serviceAccountRef` | `external-secrets-vault`, in `external-secrets` for `openbao` and without a namespace for the other two | Then the individual object: ```bash kubectl describe externalsecret -n ``` | Cause | Fix | | --- | --- | | The KV path does not exist (`bao kv get kv/` returns 404) | Store the secret first | | `property:` matches no key in the KV entry | `bao kv get kv/` lists them | | `permission denied` from `openbao-apps` | The path is outside `kv//`, or the namespace has no `external-secrets-vault` ServiceAccount | | `using cluster store ... is not allowed from namespace` | The store's `conditions` exclude the namespace: a workload on `openbao`, or a platform namespace on `openbao-apps` | ## Pitfalls > [!WARNING] > > **Rotation does not restart anything** > > A rotated value reaches the `Secret` within `refreshInterval`, but a pod that read it into an environment variable keeps the old one until something restarts it. A rollout, or Reloader-style tooling, is the other half of rotation. # Gateway API The [Kubernetes Gateway API]() replaces the Ingress resource, implemented by Cilium for both load balancing and TLS termination. The improvement is the split in ownership: the cluster owns the `Gateway` — addresses, certificates, ports — and each app owns its own `HTTPRoute`. ## At a glance | | | | --- | --- | | Namespace | `kube-system`, where Cilium runs | | Depends on | [Cilium]() to implement it, [cert-manager]() for the wildcard certificates | | If it is down | Nothing reaches any hostname. Running pods keep running | | Health check | `kubectl -n kube-system get gateway` → both `PROGRAMMED=True` with an address | | Files | `payload/platform/gateway-api/`, `payload/platform/gateway-api-crds/` | ## Configuration Two Gateways in `kube-system`, each with a dedicated address from the Cilium L2 pool, defined in `gateways.yaml`. `apps-gateway` serves user-facing workloads under the router's public address; `infra-gateway` serves the platform UIs, LAN only. A route names a listener by `sectionName`: | Gateway | Listener | Hostnames | Certificate | | --- | --- | --- | --- | | `apps-gateway` | `https` | `*.k8s.wlkr.ch` | `apps-gateway-tls` | | `apps-gateway` | `https-wollbi` | `*.wollbi.ch`, the neighbourhood sites | `wollbi-gateway-tls` | | `apps-gateway` | `ssh` | `*.ssh.wlkr.ch`, passthrough; see [SSH listener](<#ssh-listener>) | None here | | `infra-gateway` | `https` | `*.infra.k8s.wlkr.ch` | `infra-gateway-tls` | A listener pins one hostname pattern and a wildcard certificate covers one zone, so a new domain is three files, never a name added to an existing listener: 1. A `Certificate` in `payload/platform/certificates/certificates.yaml`. 2. An HTTPS listener in `gateways.yaml` referencing it. 3. The zone in `domainFilters` in `payload/platform/external-dns/application.yaml`, or its routes are silently ignored. The HTTPS listeners terminate TLS with cert-manager's wildcard certificates, so a new hostname needs no certificate of its own and nothing to renew. Each HTTPS listener pins its hostname pattern, so a route can only claim a name inside its Gateway's tree. A wildcard is a suffix match, so `*.k8s.wlkr.ch` still admits an `infra` name; such a route gets no traffic, because the infra names resolve to the other Gateway and the apps certificate covers one label only. Port 80 is accepted from all namespaces only so the central rule in `http-redirect.yaml` can send it to HTTPS. `apps-gateway` carries the annotation that makes [external-dns]() publish the router's public address instead of the Gateway's LAN one; the router forwards ports 80 and 443 to that LAN address. `payload/platform/cilium/rbac-gateway-fix.yaml` grants the operator read access to Ingresses and IngressClasses: its Gateway API controller lists them too, and the chart grants that only with the Ingress controller enabled. Retry removing it on a chart major. ### Backend protocol Envoy speaks HTTP/1.1 to every backend unless the Service port says otherwise: Cilium's `gatewayAPI.enableAppProtocol` makes it honour `appProtocol: kubernetes.io/h2c` and open HTTP/2 cleartext instead. A `GRPCRoute` needs that, because the Gateway translates gRPC-web into native gRPC, which only an HTTP/2 backend answers; Argo CD's `http2` port is the one user, see [Argo CD → CLI access](). The CRDs arrive twice from one tag: `make install-cilium` applies the release bundle `standard-install.yaml` before ArgoCD exists, and the `gateway-api-crds` Application applies `config/crd/standard`, the directory that bundle is generated from. The Makefile reads the tag out of the Application, so the two cannot drift apart. ## Usage Where the chart renders an `HTTPRoute` (Argo CD, OpenBao, Grafana), enable it in the values; a route only becomes a file of its own when it targets something the chart does not own, such as the Authentik outpost. Otherwise create an `HTTPRoute` in the app's namespace referencing the right Gateway; [external-dns]() creates the record and the wildcard covers the name: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app namespace: my-app spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: apps-gateway namespace: kube-system sectionName: https hostnames: - "my-app.k8s.wlkr.ch" rules: - backendRefs: - name: my-app-svc port: 80 ``` Use `infra-gateway` with a `*.infra.k8s.wlkr.ch` hostname for platform tools, and the listener from the table above whose hostnames cover the name. ### SSH listener The in-cluster Claude Code agents are reached over SSH tunnelled in TLS on port 443, the only port the router forwards to `apps-gateway`. Each agent's chart ships a `TLSRoute` for `.ssh.wlkr.ch` pointing at its SSH Service. | Setting | Why | | --- | --- | | `mode: Passthrough` | Cilium rejects a `TLS` listener in `Terminate` mode: the listener goes `Accepted=False` with "Using TLSRoute with TLS.mode Terminate is unsupported", and no route attaches. The Gateway forwards the stream by SNI and the agent pod ends the TLS with its own certificate | | No `Certificate` here | A Secret cannot be referenced across namespaces, and a passthrough listener has no use for one. The chart requests one per session from `letsencrypt-prod` in the agent's namespace | | `allowedRoutes.kinds: TLSRoute` | The listener serves nothing else, and `HTTPRoute`s cannot attach to it by accident | | `TLSRoute` `v1` | Installed in the standard channel of the Gateway API CRDs; `v1alpha2` is no longer served. Cilium reads the CRD once at operator start, so after the CRDs change, restart `cilium-operator` | DNS needs no record by hand: [external-dns]() reads the `TLSRoute` hostnames and publishes the router's public address, as for an `HTTPRoute`. A client connects with a TLS tunnel as its `ProxyCommand`, for example `openssl s_client -quiet -connect %h:443 -servername %h`. Envoy checks its egress against the backend like any Gateway traffic, so the agents' namespace must admit the `ingress` entity on the SSH port; see [Security Policies](). ## Health check Check that each HTTP listener carries only the redirect — `http` should show exactly `1`: ```bash kubectl get gateway infra-gateway -n kube-system \ -o jsonpath='{range .status.listeners[*]}{.name}={.attachedRoutes}{"\n"}{end}' ``` ## Pitfalls > [!CAUTION] > > **Always set `sectionName`** > > A route that names no listener attaches to both, and on port 80 it beats the redirect: Gateway API resolves competing routes by hostname specificity, so a route naming a hostname wins over the catch-all redirect and serves the app in cleartext. Nothing reports it — every route is `Accepted`. > [!WARNING] > > **Nothing stops two apps claiming the same hostname** > > Both Gateways admit routes from every namespace, so a stray `HTTPRoute` can attach itself to either Gateway and claim a name inside its hostname pattern — see [Security Posture](). # GitHub runners Linux jobs of a private repository run here instead of on GitHub's hosted runners, whose minutes a free organisation pays for. The [Actions Runner Controller]() is two components: `arc-controller` runs the controller and its CRDs, `arc-runners` one scale set per repository. A job gets a fresh runner pod that exits when the job ends. macOS jobs cannot run here and stay on GitHub. ## At a glance | | | | --- | --- | | Scale sets | `homelab` and `homelab-wait` on `somuch-cloud/extractor.cafe` | | Workflow label | `runs-on: homelab` | | Depends on | [OpenBao]() for the GitHub App in `kv/arc-runners/github-app` | | If it is down | Jobs on `runs-on: homelab` queue on GitHub until a runner registers; nothing else notices | | Health check | `kubectl -n arc-systems get pods` → the controller and one listener per scale set Running | | Namespaces | `arc-systems` for the controller and the listeners, `arc-runners` for the runner pods, `arc-jobs` for the job pods the container hook starts | | Files | `payload/platform/arc-controller/`, `payload/platform/arc-runners/`, and the three namespaces' files under `payload/platform/security/` | ## Configuration | Setting | Why | | --- | --- | | Controller and scale set chart on the same `targetRevision` | The controller deletes an `AutoscalingRunnerSet` whose chart version differs from its own, and Argo CD recreates it, in a loop; Renovate moves both in one PR | | `githubConfigUrl` on the repository, not the organisation | No other repository in the organisation can schedule code onto the cluster | | `githubConfigSecret: github-app` | A GitHub App rather than a PAT: no expiry and a token scoped to the installed repository — see [GitHub App](<#github-app>) | | `controllerServiceAccount` spelled out | The chart finds the controller with Helm `lookup`, which Argo CD does not run | | `minRunners` and `maxRunners` | No idle pods between pushes; the ceiling lets several pull requests run their parallel Linux jobs at once without queueing, while the requests stay a fraction of the workers' memory | | `containerMode: kubernetes-novolume` | A job with `services:` or `container:` gets its own pod, started by the container hook through the API server. No privileged Docker daemon and no PVC per job. Where that pod runs, and with which token: [Job pods](<#job-pods>) | | `ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER: "false"` | A job without `container:` runs in the runner pod itself, as on a hosted runner | | `arc-runners` and `arc-jobs` enforce `baseline` | The runner image uses `sudo`, and a service container such as PostgreSQL starts as root; neither passes `restricted` | | `arc-jobs` is exempt from the user-namespace kill | buildah builds the images as root without `CAP_SYS_ADMIN`, which `baseline` forbids, so it re-executes itself in a user namespace; [Tetragon]() records those instead of killing them | | The runner image as a full reference with a digest | Renovate tracks it; the chart passes the same image to the hook for the job pod's init container | | `resources` | Unmeasured; the request fits a test suite with a database beside it, and the limit stops a runaway build before it starves the node | | A second scale set, `homelab-wait`, for `ci-ok` alone | The wait holds its runner until every other check on the commit has finished; on the shared set, as many open pull requests as `maxRunners` fill it with waits and the checks they wait for never start. It runs no build, so it has no container mode and a small request | ### Job pods The chart's own ServiceAccount for `kubernetes-novolume` may read every Secret in `arc-runners`, where the GitHub App key and each runner's JIT config live, and a step without `container:` runs in the runner pod with that token. The `homelab` set therefore runs its hook one namespace over. | Setting | Why | | --- | --- | | `serviceAccountName: homelab-runner` | The chart then renders no Role of its own; `job-pods-rbac.yaml` grants the ServiceAccount rights in `arc-jobs` only | | `ACTIONS_RUNNER_KUBERNETES_NAMESPACE: arc-jobs` | The hook creates the job pod and its environment Secret there, so the Secret verbs it needs reach nothing else | | The Role is the hook's `requiredPermissions`, Secret `get` and `list` included | The hook checks every verb before each command and fails the job if one is missing; a step can still read the environment of another job running at the same time, from the same repository | | `default` ServiceAccount in `arc-jobs` without a token | A job pod never calls the API server | All jobs share `arc-jobs` and one Role, so a job can reach the pods and Secrets of any job running beside it. That is accepted because no repository with other contributors will use these runners; if that changes, it takes one scale set and namespace per trust level, each with its own Role. ### Controller The `arc-controller` values, and why: | Setting | Why | | --- | --- | | `flags.watchSingleNamespace: arc-runners` | The controller gets a Role in this namespace instead of a ClusterRole over every namespace | | `logLevel: info`, `logFormat: json` | The chart's default is `debug`, which logs every reconcile; JSON is what Loki parses | | Restricted security context on the controller and the listener | Both are distroless and run as non-root; the namespace enforces `baseline` and audits `restricted` | ### Metrics `metrics` in the controller values turns on `/metrics` on port 8080 of the controller and of every listener it starts. The listener's are the scale set's view from GitHub (assigned jobs, desired, idle and registered runners, job start-up and run times), the controller's the runner pods it manages. Both PodMonitors in `payload/platform/arc-controller/podmonitors.yaml` set `honorLabels`, so the metrics keep their own `namespace`, the runners' namespace, instead of the pod's `arc-systems`. The **GitHub Actions Runners** dashboard in `payload/platform/arc-controller/grafana-dashboards.yaml` reads them. The job metrics carry no scale set label, so its job panels take the scale set from the listener pod's name. Upstream's sample dashboard is not vendored: it filters on `actions_github_com_scale_set_name`, which this version does not export, and most of its panels stay empty. ### Network policy Every rule in `arc-runners.yaml` and `arc-jobs.yaml` under `payload/platform/security/network-policies/`. Unlike the other namespaces, egress is every public address rather than a list of names. | Rule | Why | | --- | --- | | Ingress from the namespace and the node only | Runners and job pods serve nothing; the listener in `arc-systems` reaches GitHub, not the runners, and a runner reaches its job pod through the API server | | Egress to the API server, `homelab` runner pods only | The container hook creates the job pod and execs each step into it; the `homelab-wait` runners run no hook, and job pods carry no token | | Egress to `0.0.0.0/0` on 80 and 443, minus the private, carrier-grade and link-local ranges | A job downloads whatever its workflow names (actions, toolchains, packages), which no name list keeps up with. Cutting the private ranges keeps the LAN, the nodes and every in-cluster service out of reach | The controller's and listeners' egress, GitHub's API and the Actions service only, and Prometheus' scrape of their metrics port, are in `payload/platform/security/network-policies/arc-systems.yaml`. ### GitHub App One app per organisation, installed on each repository that has a scale set. 1. Create the app under the organisation's **Settings → Developer settings → GitHub Apps → New GitHub App**: any name and homepage URL, **Webhook → Active** unticked, **Repository permissions → Administration: Read and write**, **Metadata: Read-only**, installable on this account only. 2. On the app's page, note the **App ID** and **Generate a private key**; the `.pem` file downloads. 3. **Install App** → **Only select repositories** → the repository. The installation ID is the number at the end of the installation's URL. 4. Store all three: ```bash make bao-secrets # answers the arc-runners/github-app prompts ``` 1. Push the secret through and wait for the listener: ```bash kubectl -n arc-runners annotate externalsecret github-app force-sync="$(date +%s)" --overwrite kubectl -n arc-systems get pods -l app.kubernetes.io/component=runner-scale-set-listener ``` The scale set then appears under the repository's **Settings → Actions → Runners**. ## Usage A job opts in with its label: ```yaml runs-on: homelab ``` | Job shape | What runs where | | --- | --- | | Steps only | In the runner pod, on the runner image | | `container:` | A job pod from that image; steps exec into it | | `services:` | Needs a `container:` too; the services join the job pod and answer on `localhost` | The runner image is GitHub's minimal [`actions-runner`](), not the hosted `ubuntu-latest`: `git`, `curl`, `jq`, `unzip` and `sudo`, and no language toolchains. A job installs what it needs, pinned, or uses a `setup-*` action. ### Adding a scale set 1. Install the [GitHub App](<#github-app>) on the repository. 2. In `payload/platform/arc-runners/application.yaml`, copy a chart source with its own `helm.releaseName`, `githubConfigUrl` and `runnerScaleSetName`; a set that runs job pods keeps `containerMode`, `serviceAccountName` and `ACTIONS_RUNNER_KUBERNETES_NAMESPACE`. 3. In `job-pods-rbac.yaml`, a ServiceAccount, Role and binding for the set's hook, or `homelab-runner` again when the jobs share its trust level, see [Job pods](<#job-pods>). 4. In `payload/platform/security/network-policies/arc-runners.yaml`, the `container-hook` policy selects `actions.github.com/scale-set-name` `homelab` only; a set that runs a hook needs the same rule for its name. 5. A repository that runs `ci-ok` needs a `-wait` twin of the set, like `homelab-wait`, for the reason in [Configuration](<#configuration>). 6. Workflows opt in with `runs-on: `. ## Health check ```bash kubectl -n arc-runners get autoscalingrunnerset,ephemeralrunner kubectl -n arc-systems get pods kubectl -n arc-systems logs -l app.kubernetes.io/component=runner-scale-set-listener --tail=50 ``` A job waiting on `runs-on: homelab` with no runner pod in `arc-runners` means the listener is not getting it; its log and the controller's say why. ## Pitfalls - **`Job Container is required.`** A job with `services:` and no `container:`. The hook runs services only beside a job container. - **No Docker daemon.** The image carries the `docker` CLI, but nothing answers it: `docker build` and `docker run` steps fail. - **No listener until the app is in OpenBao.** An empty `kv/arc-runners/github-app` syncs cleanly; the controller then fails to register the scale set with GitHub and says so in its log. - **Removing a scale set takes the controller.** Its Roles carry the `actions.github.com/cleanup-protection` finalizer, which only the controller removes; take the scale set out first and let it go before touching `arc-controller`. # Kured [Kured]() applies the updates the nodes have staged. It watches every node for a sentinel file and, when it finds one, takes a cluster-wide lock, cordons and drains the node, reboots it and uncordons it when it is back. Flatcar's own `locksmithd` is masked because it reboots without draining, on a cluster where every node is a storage node. ## At a glance | | | | --- | --- | | Namespace | `kured` | | Depends on | [Monitoring]() for the alerts it gates on | | If it is down | Nothing visible: staged OS and sysext updates simply never get applied | | Health check | `kubectl -n kured logs -l app.kubernetes.io/name=kured --tail=50` | | Files | `payload/platform/kured/application.yaml` | ## Configuration An update stages on disk and signals it by touching `/run/reboot-required`, Kured's default sentinel. `/run` is tmpfs, so the marker clears itself on reboot. | Mechanism | Stages | Signals by | | --- | --- | --- | | Flatcar OS (update-engine) | New image on the passive A/B partition | `flatcar-reboot-sentinel.timer` | | Kubernetes sysext | New `.raw` under `/opt/extensions/kubernetes` | `systemd-sysupdate` drop-in | ``` flowchart TD OS["update-engine
OS image on the passive partition"] --> SEN KUBE["systemd-sysupdate
kubernetes sysext"] --> SEN SEN["/run/reboot-required
sentinel, on tmpfs"] --> TICK{"Kured checks
every period"} TICK -->|"no sentinel"| SLEEP["Nothing to do until
an update stages one"] TICK -->|"sentinel present"| ALERT{"Blocking alert firing?
Ceph, etcd, node, API, Velero"} ALERT -->|"yes"| HELD["Reboot deferred while
the cluster is unhealthy"] ALERT -->|"no"| LOCK{"Cluster lock free?"} LOCK -->|"another node
is rebooting"| TAINT["Taint PreferNoSchedule,
wait for the lock"] LOCK -->|"acquired"| DRAIN["Cordon and drain"] DRAIN -->|"drain fails"| GIVEUP["No reboot: uncordon,
release the lock, retry"] DRAIN -->|"drained"| BOOT["systemctl reboot;
new partition and
sysexts take effect"] BOOT --> UP["Node comes back,
sentinel gone with tmpfs"] UP --> DONE["Uncordon, drop the annotation,
release the lock after a delay"] style SEN stroke-width:3px ``` The values are in `payload/platform/kured/application.yaml`; the reasoning: | Setting | Why | | --- | --- | | No reboot window | Kured acts as soon as it sees a sentinel; the guards below make that safe, not the clock | | `concurrency` and `lockReleaseDelay` | One node down at a time. The rebooted node is uncordoned as soon as Kured's pod is back; the delay only holds the *next* node, long enough for the OSD to rejoin and the few minutes of backfill a reboot leaves, and as long as `CephOSDDown`'s `for`, so an OSD that did not come back is a firing alert before the next drain | | `forceReboot: false` | A node that will not drain is a node worth looking at | | `preferNoScheduleTaint` | A node pending reboot stops attracting pods about to be evicted again | | `annotateNodes: true` | `kubectl get node -o yaml` shows why a node is cordoned | | `VeleroBackupInProgress` on the alert list | Velero marks any backup it finds running at startup as Failed, and draining its node restarts it. The rule in `payload/platform/velero/application.yaml` compares attempts with outcomes per live Velero pod over a window longer than Velero's default limit on asynchronous operations; a backup that hangs drops out of the window, so it holds reboots for that long at most. A backup that starts in a Velero pod's first scrape interval has no zero sample to increase from and does not hold the gate. It routes to the `null` receiver because it fires every night | | `alertFilterRegexp` with `alertFilterMatchOnly: true` | Kured asks Prometheus before taking a node down and refuses while any Ceph, etcd, node or API alert on the list fires. This is the automated "confirm Ceph has recovered" step from [Rebooting a node](): a second reboot during a backfill can take a placement group below its minimum replica count. Kured matches the alert name only, so Prometheus loads rules from [platform namespaces only]() | > [!NOTE] > > **The regex means the opposite of what it looks like** > > `alertFilterRegexp` normally lists alerts to **ignore**. With `alertFilterMatchOnly: true` these become the only alerts that block, because something is almost always firing in a homelab and blocking on any alert would mean never rebooting. An alert not on the list does not stop a reboot, so widen it if something turns out to matter — and read the flag twice before editing it. A name that no `PrometheusRule` defines blocks nothing and fails silently; the list once carried eight such names. Check every entry against the rules that exist, with the last command under [Health check](<#health-check>). ## Usage To stop Kured rebooting anything without uninstalling it, take its cluster lock by hand; it holds until the annotation is removed. Clearing the sentinel does not work, because `flatcar-reboot-sentinel.timer` in `ansible/templates/butane_node_config.yaml.j2` re-touches it while an update is staged, and a DaemonSet cannot be scaled. ```bash kubectl -n kured annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}' kubectl -n kured annotate ds kured weave.works/kured-node-lock- # release it ``` To force a node to reboot on the next pass: ```bash ssh core@ sudo touch /run/reboot-required ``` `NodeRebootPending` in `payload/platform/monitoring/node-update-rules.yaml` fires when the sentinel has been there for two days: Kured reboots one node at a time and refuses while Ceph or etcd is unhealthy, so a marker that old is a blocked node, not a queued one. ## Health check ```bash # Which nodes want a reboot: kured_reboot_required is the sentinel's presence, per node kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 & curl -s localhost:9090/api/v1/query --data-urlencode 'query=kured_reboot_required == 1' \ | jq -r '.data.result[].metric.node' # Which node is being rebooted right now kubectl get nodes -o json | jq -r '.items[] | select(.metadata.annotations["weave.works/kured-reboot-in-progress"]) | .metadata.name' # Whether a node has staged anything ssh core@ 'ls -l /run/reboot-required; update_engine_client -status' kubectl -n kured logs -l app.kubernetes.io/name=kured --tail=50 # The alert names the gate may use: every entry in alertFilterRegexp must be here kubectl get prometheusrule -A -o json | jq -r '.items[].spec.groups[].rules[] | select(.alert) | .alert' | sort -u | grep -E 'Ceph|etcd|KubeNode|KubeAPI|VeleroBackupInProgress' ``` # Logging [Loki]() stores logs; [Grafana Alloy]() collects them. Both are queried from the existing Grafana at [monitoring.infra.k8s.wlkr.ch](). Without them a crashed pod's logs are gone with the container, and on Flatcar the `kubelet`, `containerd`, `systemd-sysupdate` and `update-engine` logs live in journald and are reachable only over SSH to the node that is misbehaving. ## At a glance | | | | --- | --- | | Namespace | `logging` | | Depends on | [Rook-Ceph]() object storage for chunks, [Monitoring]() for the Grafana that queries it | | If it is down | Logs stop being collected and are not backfilled. The [audit log]() loses its durable copy, and the [log alerts](<#alerting>) stop | | Health check | `kubectl -n logging get pods`, then a `{job="kubernetes-audit"}` query in Grafana | | Metrics | Loki and Alloy, each through its chart's `ServiceMonitor`; both sets of dashboards are listed under [Monitoring]() | | Files | `payload/platform/loki/` and `payload/platform/alloy/`, one chart Application each (`loki`, `alloy`); `payload/platform/logging/`, the `logging` Application: the bucket claim in `bucket.yaml`, `loki-rules.yaml`, the Grafana datasource and the Alloy dashboards | > [!NOTE] > > **Not Promtail, and not the `grafana/loki` chart** > > Promtail reached end of life in March 2026; Alloy is its supported replacement. Loki comes from `grafana-community/loki`: the OSS chart moved to [grafana-community/helm-charts]() in March 2026, and what stayed in `grafana/loki` is maintained for Grafana Enterprise Logs customers, pins a `k8s-sidecar` several minor releases behind and closes open-source pull requests with a pointer to the community repository. ## Configuration ### Sources | Source | Path | Labels | | --- | --- | --- | | Container logs | `/var/log/pods/__//*.log` | `namespace`, `pod`, `container`, `node`, `app` | | Node journal | `/var/log/journal` | `unit`, `node`, `job="systemd-journal"` | | API server audit log | `/var/log/kubernetes/audit/audit.log` | `verb`, `audit_level`, `node`, `job="kubernetes-audit"` | | Hubble policy verdicts | `/var/run/cilium/hubble/events.log` | `verdict`, `node`, `job="hubble"` | The first three are on the root filesystem, so one `varlog: true` mount covers them; the Hubble export is a separate read-only hostPath — see [Cilium](). The audit log exists only on the control-plane nodes; Alloy runs everywhere and `local.file_match` finds nothing on the workers. Audit events keep only `verb` and `audit_level` as labels — small closed sets — while user and resource stay in the line where `| json` can reach them. See [Audit logging]() for what is recorded. ### Alloy pipeline | Stage | Why | | --- | --- | | `discovery.kubernetes` with a `spec.nodeName` field selector | Each agent discovers only its own node's pods; without it every agent watches every pod and discards all but its own | | `stage.cri {}` on container logs | containerd writes ` `; without it the timestamp lands inside the line and Loki stamps everything at ingest time | | `discovery.relabel "journal"` sets `job` | `loki.source.journal` overwrites `job` with its component ID after applying `labels`; only relabel rules run after that | | `stage.json`, `stage.labels`, `stage.timestamp` on audit events | Stamps each event with its `requestReceivedTimestamp` rather than when Alloy read the file | | Setting in `alloy/application.yaml` | Why | | --- | --- | | Control-plane toleration | Without it Alloy runs on the workers alone and neither the audit log nor the control-plane kubelets' journals reach Loki | | `enableReporting: false` | Alloy otherwise sends usage statistics to `stats.grafana.org`; the policy audit showed the call, and telemetry is off in every component here | | `runAsUser: 0` | The log files are root-owned; a non-root Alloy does not fail, it silently collects nothing. Read-only root, no capabilities and `RuntimeDefault` seccomp lock the rest down | | `emptyDir` at `/tmp` | The chart's `storagePath` defaults to `/tmp/alloy`, unwritable on a read-only root; it holds tailing positions, which did not survive a restart before either | | `rbac.rules` without `secrets` | The chart's ClusterRole reads every Secret for `remote.kubernetes.secret`, which nothing here uses; [`KubernetesAuditSecretReadByUser`](<#alerting>) would show such a read, but a grant with no purpose is better not made | ### Storage Loki runs as a single binary (`deploymentMode: Monolithic`) and keeps chunks in the [Ceph object store](); a local PVC on `rook-ceph-block` holds the WAL and the index being built. The bucket is the `ObjectBucketClaim` in `logging/bucket.yaml`, with a fixed `bucketName`: a generated name carries a random suffix that would have to be read back at runtime, a fixed one keeps Loki's config static and identical after a rebuild. | Setting in `loki/application.yaml` | Why | | --- | --- | | `singleBinary.extraArgs` `-config.expand-env=true` and `singleBinary.extraEnvFrom` | Rook writes the bucket credentials to a Secret, Loki reads them as `${AWS_ACCESS_KEY_ID}` — nothing in Git | | `loki.analytics.reporting_enabled: false` | Loki otherwise sends usage statistics to `stats.grafana.org` a few times a day; same reason as Alloy's switch above | | `chunksCache` and `resultsCache` off | Four memcached pods in front of a Loki this size | | `loki.podSecurityContext` | Adds `RuntimeDefault` seccomp at pod level, covering the rules sidecar too | | Retention with the compactor enabled | Old chunks are actually deleted | | `rulerConfig` | The ruler evaluates LogQL alerts and posts them to Alertmanager; without it the audit log and the journal are query-only. Rules come from local files the chart's sidecar copies out of `loki_rule` ConfigMaps, so `storage.type` is `local` rather than the bucket the chart would pick; `rule_path` is the ruler's scratch directory and must not be the rules directory | | `rbac.namespaced: true` and `sidecar.rules.resource: configmap` | The rules are ConfigMaps in `logging`, so a Role there replaces the chart's cluster-wide ClusterRole, for the same reason as Alloy's `rbac.rules` above; the chart's binding could not be repointed because a `roleRef` is immutable. `resource: configmap` keeps the sidecar from listing Secrets | | `retention_stream` for the `findings-history` container | The daily [Trivy summary]() is a few lines and the only long-range record of findings, so that one stream outlives the general `retention_period` by far | ### Alerting Alerts over logs live in `logging/loki-rules.yaml`, a ConfigMap labelled `loki_rule` in the same format as a `PrometheusRule` group with LogQL expressions. The `k8s-sidecar-target-directory: fake` annotation puts the file under the tenant directory the ruler reads (`fake` is the tenant when `auth_enabled` is off). Prometheus-side rules stay with their components; these are the events only a log carries. Loki's own health alerts (request errors, latency, panics, the compactor) are the chart's `monitoring.alerts`, the mixin Grafana maintains, not rules written here. | Alert | Fires on | Why it is a log alert | | --- | --- | --- | | `KubernetesAuditExecIntoPod` | A successful `exec` or `attach` | Only the [audit log]() records who ran what in which pod | | `KubernetesAuditSecretReadByUser` | A Secret read by a person, or by a ServiceAccount outside its own namespace that is not on the allowlist of platform controllers | A stolen token reads across namespaces; the allowlist is the ServiceAccounts RBAC grants Secrets to cluster-wide, plus the runner's in `arc-jobs` only, where its [container hook]() keeps each job's environment, and a ServiceAccount reading its own namespace is what its Role is for | | `KubernetesAuditRbacBindingChanged` | A ClusterRole or ClusterRoleBinding written by anything but the ArgoCD controller or a `kube-system` controller | Argo CD is the only intended writer; namespaced RoleBindings are left out because CloudNativePG reconciles one per database continuously | | `KubernetesAuditForbiddenBurst` | More than ten 403s from one identity in ten minutes | What RBAC probing from a compromised pod looks like | | `KubernetesAuditAnonymousRequest` | A successful anonymous request | The audit policy drops the health endpoints, the only legitimate anonymous paths | | `KubernetesAuditEsoTokenMinted` | A successful `serviceaccounts/token` create for an `external-secrets-vault` ServiceAccount by anyone but ESO's controller, a kubelet starting a pod under that ServiceAccount included | OpenBao records the same role and ServiceAccount whoever minted the token, so only the audit log names the minter. A stolen controller token or a legacy token Secret does not trip it: neither shows a different user on this request | | `OpenBaoSecretReadOutsideEso` | A `kv/data/` read from OpenBao by any token whose display name is not `kubernetes--external-secrets-vault`, ESO's login for one of its [stores]() | [OpenBao's audit device]() writes to stdout; ESO is the only day-to-day reader, every Kubernetes role is bound to that ServiceAccount name, and a token created by hand is prefixed `token-`, so it cannot borrow the name | | `TetragonSudoInContainer` | `sudo` executed in a container, except the `smartctl` and `nvme` calls `ceph-osd` makes once a day for its device health report | Only the exec event carries the binary and its parent | | `TetragonPrivilegedExec` | A setuid, setgid or file-capability binary raised privileges on exec, with the same `ceph-osd` exception | The `binary_properties` field exists only in the exec event | | `TetragonFilelessExec` | A binary executed from memory, via `memfd_create` or an anonymous descriptor, except runc's sealed self-copy on every `exec` | Same | | `TetragonDeletedBinaryExec` | A binary executed after its last link was removed | Same; see [Tetragon]() | | `TetragonUnsignedKernelModule` | A kernel module loaded without a valid signature, on a node or from a container | The signature result is an argument of the kprobe event, not a metric label | | `HubblePolicyDrops` | More than three `POLICY_DENIED` packets in ten minutes for one source workload, destination and port, each counted once although Hubble also reports it as a policy verdict; replies, ICMP errors (a pod's port-unreachable answer to a DNS reply that came too late), pods still in the `init` identity and addresses with no pod behind them are left out. The `namespace` label is the policy that refused, so the mail names the file to edit | A connection's retransmits pass the threshold and a name rule's single early `SYN` does not; the metric counts per namespace pair only and misses the first burst of a new pair | | `NodeSshLogin` | An accepted SSH login on a node | Nothing routine logs in after provisioning | | `NodeSshAuthFailures` | More than five failed SSH attempts on a node in ten minutes | Password authentication is off; repeats are a scan or a retried key | To confirm the ruler loaded a rule: ```bash kubectl -n logging port-forward svc/loki 3100:3100 curl -s localhost:3100/loki/api/v1/rules ``` ### Dashboards `grafana-dashboards.yaml` carries three dashboards from the Alloy mixin (`operations/alloy-mixin/rendered/dashboards/`), unmodified; the chart renders none and Renovate does not see the copy, so re-copy them when Alloy moves a minor version. The mixin's other dashboards cover features this Alloy does not use. The Loki datasource in `grafana-datasource.yaml` declares its namespace as `monitoring`, the only namespace the Grafana sidecar watches for `grafana_datasource` ConfigMaps. ## Usage Choose **Loki** as the datasource in Grafana and query by label: ```text {namespace="rook-ceph"} |= "error" {unit="kubelet.service", node="odin"} {namespace="openbao"} |= "sealed" {job="kubernetes-audit"} | json | objectRef_resource="secrets" ``` ## Health check ```bash kubectl -n logging get pods ``` Then run a `{job="kubernetes-audit"}` query in Grafana; an empty result with healthy pods means the control-plane agents are not reading the file. ## Pitfalls > [!WARNING] > > **Loki settings belong on `singleBinary`, not `global`** > > `global.extraArgs` and `global.extraEnvFrom` look right, but the single-binary StatefulSet reads only `singleBinary.extraArgs` and `singleBinary.extraEnvFrom`. Set globally, the config is full of unexpanded `${...}` and Loki cannot authenticate to RGW, with an error pointing nowhere near the cause. Likewise `singleBinary.podSecurityContext` does not exist and is silently inert. # metrics-server [metrics-server]() serves the `metrics.k8s.io` API that `kubectl top`, every `HorizontalPodAutoscaler` and the [descheduler]() read; the scheduler does not, it places on requests. Without it `kubectl top` errors and an HPA sits at `/` with no logs, events or clue. Prometheus is not a substitute: it answers PromQL from its own store, while the resource metrics API is an aggregated Kubernetes API that core controllers read directly. ## At a glance | | | | --- | --- | | Namespace | `kube-system`; kubelet-csr-approver in `kubelet-csr-approver` | | Depends on | kubelet-csr-approver, or the kubelet certificates it verifies are never issued; [cert-manager]() for its own serving certificate | | If it is down | `kubectl top`, every HPA and the [descheduler]()'s node usage. Nothing else notices | | Health check | `kubectl top nodes` → a row per node | | Files | `payload/platform/metrics-server/`, `payload/platform/kubelet-csr-approver/` | ## Configuration ### Verifying the kubelet instead of trusting it A kubelet serves a self-signed certificate by default, so the usual shortcut is `--kubelet-insecure-tls`, which makes the metrics path spoofable by anything that can occupy a kubelet's address. Kubernetes deliberately does not auto-approve `kubernetes.io/kubelet-serving` CSRs, since a compromised node could request a certificate for any name, so doing it properly takes three pieces: | Piece | Where | What it does | | --- | --- | --- | | `serverTLSBootstrap: true` | `ansible/templates/kubeadm.yaml.j2` | The kubelet requests a serving certificate from the cluster CA | | [kubelet-csr-approver]() | `payload/platform/kubelet-csr-approver/application.yaml` | Approves those CSRs, constrained by `providerRegex` (only the inventory's node names; anything else is **denied**), `providerIpPrefixes` (the nodes' subnet), a one-day `maxExpirationSeconds`, and `bypassDnsResolution` because node names come from the Ansible inventory, not a resolver | | `--kubelet-certificate-authority` | `payload/platform/metrics-server/application.yaml` | metrics-server verifies against the cluster CA | | Setting in `metrics-server/application.yaml` | Why | | --- | --- | | Flags via `defaultArgs`, not `args` | The default list already sets `--kubelet-preferred-address-types`, and a flag passed twice leaves the winner to parsing order. It is `InternalIP` because node hostnames do not resolve here | | Two replicas and a PodDisruptionBudget | A node reboot does not take the resource metrics API down | | `tls.type: cert-manager` with `apiService.insecureSkipTLSVerify: false` | The other half of the path: the chart's self-signed `Issuer` signs the serving certificate and the cainjector copies its CA into the `APIService`, so kube-apiserver verifies metrics-server instead of trusting whatever answers at the Service | ## Health check ```bash kubectl get csr | grep kubelet-serving # should be Approved,Issued kubectl get apiservice v1beta1.metrics.k8s.io # Available True, caBundle filled kubectl top nodes ``` `Pending` CSRs mean kubelet-csr-approver is not running or a node name does not match the regex. `Denied` means the regex or the IP prefix is wrong — check the controller's logs before widening either. ## Pitfalls > [!WARNING] > > **Adding a node means editing the regex** > > A node missing from `providerRegex` has its CSR denied, keeps its self-signed certificate, and silently goes missing from `kubectl top`. Add it to the regex when you add it to the inventory. # Monitoring Full observability stack based on [kube-prometheus-stack](): Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics. node-exporter runs in its own `node-exporter` namespace so that only it is admitted at the `privileged` level ([security policies]()). The chart's default rule set is good; its default routing swallows every alert, which is what most of this page is about. ## At a glance | | | | --- | --- | | Namespace | `monitoring` | | Depends on | [Rook-Ceph]() for Prometheus and Grafana volumes, [Authentik]() for the Grafana login | | If it is down | No alerts and no metrics — and [Kured]() loses the alert gate it checks before rebooting a node | | Health check | `kubectl -n monitoring get prometheus,alertmanager`; the `Watchdog` alert should always be firing | | UI | `monitoring.infra.k8s.wlkr.ch` (Grafana), `prometheus.` and `alertmanager.` | | Files | `payload/platform/monitoring/`, `payload/platform/prometheus-operator-crds/` | ## Configuration ### CRDs The `monitoring.coreos.com` CRDs are a separate `prometheus-operator-crds` Application, from the prometheus-community chart of that name, and this chart runs with `crds.enabled: false`. Cilium, cert-manager, External Secrets, Rook and OpenBao all render ServiceMonitors or PrometheusRules, and a missing kind fails their sync until the CRDs exist — see [Bootstrap convergence](). The Application has no resources finalizer, so deleting it leaves the CRDs, and every ServiceMonitor, Prometheus and Alertmanager with them, in place. > [!WARNING] > > **Two versions that have to agree** > > `prometheus-operator-crds` must carry the same operator version as kube-prometheus-stack; an operator newer than its CRDs can depend on fields they do not define. Renovate bumps them separately: merge the CRD bump first, or both together — see [Maintenance](). ### Alerting Alertmanager routes to email. The chart's `config` is off (`alertmanagerSpec.useExistingSecret`) and `alertmanager-config.yaml` renders the whole `alertmanager.yaml` through ESO into `alertmanagerSpec.configSecret`, because only the password has a file form (`smtp_auth_password_file`) and the login and recipient are e-mail addresses that do not belong in Git. | Setting in `alertmanager-config.yaml` | Why | | --- | --- | | `smtp_smarthost` on port 465 | Implicit TLS, which Alertmanager selects from the port; `smtp_require_tls` stays on so a move back to 587 cannot fall to plaintext | | `smtp_from` equals the login | mailbox.org refuses a sender the login does not own | | Grouping by `namespace`, with `group_interval` and `repeat_interval` in hours and a shorter pair for `critical`; resolved notifications sent | mailbox.org caps what an account may send. One mail per alert and namespace came to hundreds a day once a node was down, the account was refused from then on, and the critical mails went unsent with the rest. A namespace's alerts usually share a cause, so one mail lists them together | | `Watchdog` routed to `null` | It fires continuously as proof the pipeline is alive; mailing it would train the recipient to filter the sender | | `VeleroBackupInProgress` routed to `null` | It fires for every nightly backup as [Kured's gate](), not as a problem | | `KubeJobFailed` disabled in the chart (`defaultRules.disabled`) and redefined in `job-rules.yaml`, bound to Jobs started in the last day | The chart's rule fires for as long as a failed Job object exists, so one old failure reads as an outage for weeks. Failed Jobs carry no failure time, so the bound counts from the start | | Inhibit rules repeated from the chart defaults | A supplied config replaces the whole document, so nothing is inherited | | Every templated value through `toJson` | A password containing `:`, `#` or a quote cannot break the rendered file | | `externalUrl` on both `prometheusSpec` and `alertmanagerSpec` in `application.yaml` | The "Source" and Alertmanager links in a mail are built from these; unset, they point at the in-cluster Service, which nothing outside the cluster can open | The three values come from `kv/monitoring/smtp` (`username`, `password`, `to`); `make bao-secrets` prompts for them, or by hand: ```bash bao kv put kv/monitoring/smtp username=... password=... to=... ``` For another provider, change `smtp_smarthost` in the template and the three values in OpenBao. ### Scrape targets Every `ServiceMonitor` and `PodMonitor` in every namespace, through the five `*SelectorNilUsesHelmValues: false` settings: the chart treats an empty selector as unset and renders `release: kube-prometheus-stack` in its place, which limits Prometheus to its own targets. `PrometheusRule`s load only from the namespaces the `apps` project cannot deploy to, matched on the `kubernetes.io/metadata.name` label the API server sets. [Kured]() gates reboots on alert names, so a workload's rule named `CephHealthError` would otherwise stop OS updates cluster-wide. A rule in a new platform namespace loads only once that namespace is on the list in `ruleNamespaceSelector`. | Target | Made scrapable by | Notes | | --- | --- | --- | | kube-scheduler, kube-controller-manager | A per-node `bind-address` patch in `ansible/templates/butane_node_config.yaml.j2` (kubeadm binds them to `127.0.0.1`) | HTTPS with a self-signed certificate, so the ServiceMonitors set `insecureSkipVerify`; the ServiceAccount token still authenticates | | etcd | A per-node `listen-metrics-urls` patch in the same template (kubeadm listens on `127.0.0.1` only) | Plain HTTP on 2381, serving only `/metrics` and `/health`; client and peer APIs stay on mTLS | | Ceph | `monitoring.enabled` on the `CephCluster` — see [Rook-Ceph]() | | | kube-proxy | Not scraped (`kubeProxy.enabled: false`) | Cilium replaces it; the default would keep `KubeProxyDown` firing forever | | CRD status fields: CloudNativePG's ObjectStores and Backups | kube-state-metrics' `customResourceState` config in the chart values, one `rbac.extraRules` entry per kind | A `StateSet` per enum field and a `Gauge` per timestamp, no more; the rule that reads a kind sits with its component, and the namespace it lives in has to be on the rule list above | | Pod labels: OpenBao's `openbao-sealed` | kube-state-metrics' `metricLabelsAllowlist` in the chart values | Exported as `label_` on `kube_pod_labels`; only listed labels, since every one adds a series per pod | > [!NOTE] > > A cluster provisioned before the kubeadm change keeps the old flags in its static pod manifests; until those nodes are reprovisioned the three control-plane targets stay down. ### Grafana | Setting | Why | | --- | --- | | `disable_login_form` and `oauth_auto_login` | Opening the URL bounces straight to Authentik and back; `role_attribute_path` with `role_attribute_strict` maps the three Grafana groups and refuses everyone else, see [Groups and roles]() | | `analytics.*` off and `disable_gravatar` | Grafana otherwise reports usage to Grafana Labs, asks grafana.com for updates and fetches every user's avatar from Gravatar; the Grafana network policy names none of those and the policy audit showed the Gravatar call. Telemetry is off in every component here | | `grafana-oidc.yaml` | The OIDC client is generated into OpenBao and read by both sides — see [Authentik]() | | `grafana.route.main` | The chart renders Grafana's `HTTPRoute`. Prometheus and Alertmanager stay in `httproute.yaml`: their routes point at Authentik's outpost, which no chart template can express | | `grafana.admin.existingSecret`, from `grafana-admin.yaml` | Left unset, the chart generates a new password on every render, so the Secret is always OutOfSync and `checksum/secret` restarts Grafana on every sync. The admin account is the [break-glass path](); never set `grafana.adminPassword` in Git | | `grafana.rbac.create: false` with `grafana-sidecar-rbac.yaml`, and `sidecar.*.resource: configmap` | Dashboards and datasources are ConfigMaps, so an own ClusterRole and binding grant those alone instead of the chart's ConfigMaps-and-Secrets pair, for the reason given with [Alloy's `rbac.rules`](); the chart's binding is dropped rather than repointed because a `roleRef` is immutable | | `Recreate` strategy | The dashboard PVC is RWO; under `RollingUpdate` the new pod waits for a volume the old pod releases only once the new one is Ready, parking on `FailedAttachVolume` | | Prometheus and Grafana memory limits well above their requests | Both grow with series count and dashboard load, and OOMKilling the thing that reports cluster health is the failure worth avoiding; the limit is a ceiling for a runaway, sized by the [2.5x-peak rule](), not a budget | | Prometheus `retention` and `retentionSize` | Whichever is hit first wins. The size cap is the floor under the volume: a series count that grows faster than planned drops the oldest blocks instead of filling the disk and crashlooping Prometheus | | Alertmanager `automountServiceAccountToken: false`, on the ServiceAccount and in `alertmanagerSpec` | Nothing binds a role to it — the operator talks to the API — so the token would only be a credential in a pod reachable from the Gateway. The operator sets the pod field itself, so the account's setting alone changes nothing | ### Dashboards Where a chart can render a dashboard, it does; only charts without one get a vendored copy, which Renovate does not update — the header of each file names its upstream tag. | Component | Dashboards | Source | | --- | --- | --- | | [Cilium]() | Cilium Metrics, Cilium Operator, four Hubble dashboards | Chart: `dashboards`, `operator.dashboards`, `hubble.metrics.dashboards` | | [OpenBao]() | OpenBao | Chart: `serverTelemetry.grafanaDashboard` | | [External Secrets]() | External Secrets Operator | Chart: `grafanaDashboard` | | [Rook-Ceph]() | Ceph Cluster, Ceph - OSD (Single), Ceph - Pools | Vendored: `rook-ceph/grafana-dashboards.yaml` | | [Alloy]() | Alloy / Controller, Alloy / Loki Components, Alloy / Resources | Vendored: `logging/grafana-dashboards.yaml` | | [Loki]() | The Loki mixin: Reads, Writes, Operational, Chunks, Retention and the rest | Chart: `monitoring.dashboards`, with `monitoring.rules` for the recording rules they read | | ArgoCD | ArgoCD | Vendored: `argocd-config/grafana-dashboards.yaml` | | [Tetragon]() | Tetragon | Written for this cluster: `tetragon/grafana-dashboards.yaml`; upstream ships none | | [GitHub runners]() | GitHub Actions Runners | Written for this cluster: `arc-controller/grafana-dashboards.yaml`; upstream's sample filters on labels ARC 0.15 does not export | | [Trivy Operator]() | Trivy Operator | Written for this cluster: `trivy-operator/grafana-dashboards.yaml` | ## Usage ### Adding a dashboard Add a ConfigMap labelled `grafana_dashboard: "1"` next to the component it describes, in a file named `grafana-dashboards.yaml` and a ConfigMap named `-grafana-dashboards`; a Secret carrying the label is not read. The sidecar watches only the platform namespaces listed in `sidecar.dashboards.searchNamespace`, so a workload cannot replace a dashboard; a component in a new namespace adds it there, and the `apps` project must not be able to deploy to it. Prefer the chart's own dashboard where it has one. A dashboard built in the UI survives restarts on the PVC but lives nowhere else: not in Git, not on a rebuilt cluster. ### Growing the Prometheus volume Kubernetes cannot expand a StatefulSet's `volumeClaimTemplate`, so raising the request in `payload/platform/monitoring/application.yaml` does not resize the claim that already exists. The operator handles its own half of that: it fails the StatefulSet update on the immutable field, logs `recreating StatefulSet because the update operation wasn't possible`, and rebuilds it orphaned so the pod survives. Only the claim is left behind. 1. Merge the size bump and wait for the Application to sync. ```bash kubectl -n argocd get app kube-prometheus-stack -o jsonpath='{.status.sync.status}' ``` 2. Patch the claim to the same size. This is the one step nothing automates. ```bash SIZE=$(awk '/prometheusSpec:/{f=1} f&&/storage: /{print $2; exit}' \ payload/platform/monitoring/application.yaml) kubectl -n monitoring patch pvc \ prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 \ --patch "{\"spec\": {\"resources\": {\"requests\": {\"storage\": \"$SIZE\"}}}}" ``` 3. The claim parks on `FileSystemResizePending` and the filesystem grows when the volume is next mounted, so the pod is restarted and Prometheus replays its WAL — a scrape gap of about a minute, not a seamless resize. Wait for the claim to report the new capacity. ```bash kubectl -n monitoring get pvc -w \ -o custom-columns=NAME:.metadata.name,CAP:.status.capacity.storage ``` If the operator has not rebuilt the StatefulSet within a sync interval, do it by hand; `--cascade=orphan` keeps the pod and the claim. ```bash kubectl -n monitoring delete statefulset \ -l operator.prometheus.io/name=kube-prometheus-stack-prometheus --cascade=orphan ``` Alertmanager and Grafana take the same two steps, with their own claim names. ### Resetting the Grafana admin password Grafana reads the password only when it creates its database, so changing `kv/monitoring/grafana-admin` later does not change what it checks. Reset it to the value in OpenBao rather than a new one: `grafana-admin.yaml` pushes the generated password with `updatePolicy: IfNotExists`, so OpenBao keeps the first value forever and a rebuilt Grafana starts from it. ```bash PASSWORD=$(kubectl -n openbao exec openbao-0 -- \ bao kv get -mount=kv -field=password monitoring/grafana-admin) kubectl -n monitoring exec deploy/kube-prometheus-stack-grafana -c grafana -- \ grafana cli admin reset-admin-password "$PASSWORD" ``` ## Health check ```bash kubectl -n monitoring get prometheus,alertmanager kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus \ -o jsonpath='{.spec.serviceMonitorSelector}' # should print {} kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093 # then open http://localhost:9093 and check Status -> Config ``` `Watchdog` proves the pipeline as far as Alertmanager and not one step further; SMTP auth, spam heuristics and an expired password are unmonitored. Send yourself a test alert once and confirm it arrives. ## Pitfalls > [!WARNING] > > **Prometheus is a single replica, Alertmanager is not** > > Alertmanager runs two replicas spread over the workers with a PDB that lets one go, so a [Kured]() drain keeps alerting up. A second Prometheus would double the memory limit and the volume in `application.yaml`, which the [2.5x-peak rule]() leaves no room for on a worker, so it stays single: its node's drain loses metrics for the restart, and Kured's alert gate fails closed until it is back. # OpenBao [OpenBao]() is the cluster's secret store, an open-source fork of HashiCorp Vault. It holds every secret the cluster consumes, surfaced as native `Secret` objects by the [External Secrets Operator](). Sealed, nothing that needs a credential works, and the failure looks like six unrelated things breaking at once. ``` flowchart LR Operator([Operator]) -->|bao CLI| Bao[(OpenBao
KV v2)] ESO[External Secrets
Operator] -->|read, write once| Bao ESO -->|create/update| KSecret[K8s Secret] App[App Pod] -->|env / volume| KSecret style Bao fill:#e1f5ff,stroke:#0288d1 ``` ## At a glance | | | | --- | --- | | Namespace | `openbao` | | Depends on | [Rook-Ceph]() for its Raft volumes. A fresh bootstrap [pauses here]() until OpenBao is initialised and unsealed | | Seal | Shamir, unsealed by hand; the share count and threshold are set in `scripts/bao-init.sh` | | If it is down — or merely sealed | No `ExternalSecret` resolves, so cert-manager cannot renew and pods that mount a materialised Secret will not start. It looks healthy from the outside | | Health check | `bao status` on every pod → `Sealed: false`; see [Health check](<#health-check>) | | UI | `vault.infra.k8s.wlkr.ch` — the one platform UI *not* behind Authentik, deliberately | | Files | `payload/platform/openbao/` | ## Configuration | Property | Value | | --- | --- | | Mode | HA, integrated Raft on a Ceph PVC per replica; the replica count is in `application.yaml` | | Audit storage | Separate PVC on `rook-ceph-block`, written by the `file` audit device — see [Audit devices](<#audit-devices>) | | TLS | Terminates at the Gateway; plain HTTP at `http://openbao.openbao.svc.cluster.local:8200` | | Chart | Upstream [`openbao/openbao-helm`](), pinned in `application.yaml` | | Setting | Why | | --- | --- | | Pod security context restated in full | The chart's `securityContext.pod` and `.container` replace rather than merge, so a block that names only what differs loses the rest. The image's `USER` is a name, so without the numeric IDs the kubelet refuses the container as unprovably non-root — on pod recreation, typically after a reboot. The full block is what lets the namespace enforce `restricted` ([security policies]()) | | `image.repository` without a registry | The chart prepends `server.image.registry` (`quay.io`) itself | | `unauthenticated_metrics_access` | `/v1/sys/metrics` otherwise wants a token the ServiceMonitor does not have. The metrics carry counts and timings, not paths or secrets, and only the namespace policy's callers and the Gateway reach port 8200 | | `retry_join` stanzas | `bao operator init` initialises one Raft cluster on one pod; these make the other replicas join it as they start. `service_registration "kubernetes"` only labels pods `active` and `standby` | | `serverTelemetry.grafanaDashboard` | Renders OpenBao's upstream dashboard (grafana.com 23725) | | `server.gateway.httpRoute` | The chart renders the UI's `HTTPRoute`, to the `active` Service so the browser never lands on a standby; the chart's warning about terminating TLS before the server concerns certificate authentication, which nothing here uses | ### Audit devices Two, declared as `audit` stanzas in the server config in `application.yaml` (this OpenBao refuses `bao audit enable` over the API): a `file` device on the audit PVC, and a second `file` device on `stdout`, which [Alloy]() ships to Loki with the rest of the container's output. Without one, who read which secret is recorded nowhere; with only one, a full PVC would make OpenBao refuse every request, because a request no enabled device can log is refused. Values are HMACed in both; paths and identities are not. The config is read at start-up, and the StatefulSet updates `OnDelete`, so a change lands when each pod next restarts — see [Unsealing after a restart](<#unsealing-after-a-restart>). `OpenBaoSecretReadOutsideEso` in [`loki-rules.yaml`]() fires on a `kv/data/` read by anything but the External Secrets Operator's roles: the operator, a generated root token, or a token that should not exist. A standby audits a request before it resolves the token, so its `please forward to the active node` rejections carry no identity; the rule drops them, and every `ClusterSecretStore` targets the `openbao-active` Service so ESO never produces one. ```text {namespace="openbao", container="openbao"} |= `"type":"response"` |= `kv/data/` | json | __error__="" ``` ### KV layout One KV v2 engine at `kv/`; every leaf is `/`, where a workload's prefix is its namespace, the only one [its store](<#kubernetes-auth>) reaches. ExternalSecrets reference `cert-manager/route53` without the `data/` prefix ESO adds itself. A path is written by exactly one of two things: a `PushSecret` beside the `ExternalSecret` that reads it, when every value is random — see [Generated secrets](<#generated-secrets>) — or `make bao-secrets` (an agent session's PAT: `make claude-session-pat`), when a value belongs to an account outside the cluster. A path never mixes the two, because a `bao kv put` replaces the whole path. | Path | Keys | Read by | | --- | --- | --- | | `authentik/config` (generated) | `secret-key`, `bootstrap-password`, the ArgoCD and Grafana client id/secret pairs | Authentik, ArgoCD and Grafana — generating the OIDC credentials up front keeps both sides of each integration declarative | | `cert-manager/route53` | `access-key-id`, `secret-access-key` | The `certificates` Application | | `external-dns/route53` | `access-key-id`, `secret-access-key` | external-dns | | `monitoring/grafana-admin` (generated) | `password` | Grafana | | `monitoring/smtp` | `username`, `password`, `to` | Alertmanager | | `kneadtime/config` | `vapid-private-key` | The Knead Time reminder service — a P-256 key in PEM that signs every push; rotating it retires every subscription taken with the old one | | `claude-agents/argocd` | `token` | The agents' Argo CD API token — [Agent account](); empty until generated | | `shot-bridge/accounts` | `lm-username`, `lm-password`, `apns-key-id`, `apns-key-p8` | The Shot Timer bridge's La Marzocco login and APNs key — [Shot Timer bridge](); the APNs keys are empty until the key is created | | `arc-runners/github-app` | `app-id`, `installation-id`, `private-key` | The GitHub App the Actions runners register with — [GitHub runners](); empty until the app is created | | `claude-/github` | `token` | One agent session's GitHub PAT, for any session name: `make claude-session-pat SESSION=` stores it, adding `DELETE=1` removes the path. Not part of `make bao-secrets`, so a session needs no platform change. Empty until typed in | | `velero/repository` (generated) | `password` | Velero, as the kopia password of every backup repository — [Repository password](); changing it orphans every volume backup | | `dependency-track/sbom-upload` | `api-key` | The SBOM upload job, [Dependency-Track](); written even when empty, so the job fails on its own rather than stalling the sync | The two `route53` leaves are separate IAM users on purpose: cert-manager only ever writes `_acme-challenge` TXT records, though its [policy]() still allows any record change, while external-dns creates and deletes A records by design, so stolen it can repoint hostnames. One shared key collapses both into "someone owns your domain". ### Generated secrets Every random value — session keys, break-glass admin passwords, OIDC client pairs — comes from an ESO `Password` generator, written into OpenBao by a `PushSecret` in the same file as the `ExternalSecret` that reads it back: `payload/platform/authentik/secrets.yaml` here, `/secrets.yaml` in the workloads repository. A fresh cluster needs no prompt for them, and a workload that adds a path needs no change to this repository. | Setting | Why | | --- | --- | | `updatePolicy: IfNotExists` | ESO checks each property before writing it and skips one that exists, so a value is written once and the next reconcile's freshly generated one is discarded | | `deletionPolicy: None` | Deleting the `PushSecret`, or its Application, leaves the value in OpenBao; the policy grants no `delete` either | | `symbols: 0`, `allowRepeat: true` | Alphanumeric only, so a value survives a URL, a `.env` line and a shell unquoted; repeats allowed because without them the generator cannot produce more than ten digits | | `encoding` or a `template` where the reader wants a format | Umami's keys are hex, Dependency-Track's KEK is base64 of 32 bytes | | One `Password` per path, its `secretKeys` the property names | A generator produces one map per call; one key per property keeps every value independent | | Sync wave `-2` in the workloads repository | Its `ExternalSecret`s sit at `-1`, and ArgoCD waits for a wave to be healthy before the next | ESO refuses to write into a path it did not create (`secret not managed by external-secrets`), which keeps it away from everything `make bao-secrets` wrote. Reading a generated value, such as the `akadmin` password: ```bash kubectl -n openbao exec openbao-0 -- bao kv get -mount=kv -field=bootstrap-password authentik/config ``` ### Operator credential No root token outlives the command that needs it. `make bao-init` ends by creating a userpass login for the scripts and revoking the root token; the password lands in `output/credentials/openbao-operator.json` (mode `0600`). The policy and login are written by `bao_create_operator` in `scripts/lib/bao.sh`. | Setting | Why | | --- | --- | | The `operator` policy: `kv/` read, write and delete, `sys/storage/raft/snapshot` read | Everything `make bao-secrets`, `make claude-session-pat`, [rotation](<#rotating-a-credential>) and a [snapshot](<#backups>) need; policies, auth methods and snapshot restores still need root | | `sys/generate-root-token/*` and `sys/decode-token` in the same policy | Since OpenBao 2.6 `bao operator generate-root` needs a token, so without these a revoked root could not be regenerated; the unseal shares still decide | | `token_ttl` and `token_max_ttl`, short | The scripts cache the token in the pod; one a dropped connection leaves behind expires on its own, and the exit trap revokes it sooner | | Userpass, not OIDC | Authentik reads its own database password from OpenBao, so an Authentik login would be a loop | | The scripts fall back to the root token in `openbao-init.json` | A cluster initialised before this keeps working, with a warning, until `make bao-rotate-root` creates the login and revokes the root token | ### Kubernetes auth ESO authenticates with a short-lived ServiceAccount JWT, which OpenBao validates against the TokenReview API (`rbac.yaml` binds `system:auth-delegator` to the `openbao` ServiceAccount) and answers with a token bound to one policy. Three `ClusterSecretStore`s in `cluster-secret-store.yaml` each log in with their own role, so what a namespace can reach follows from the store it is allowed to use: | Store | Who may use it | Logs in as | Reaches | | --- | --- | --- | --- | | `openbao` | The platform namespaces | `external-secrets-vault` in `external-secrets` | Reads every path, because Authentik reads each workload's `kv//config`; writes only `authentik/`, `monitoring/` and `velero/`, where its `PushSecret`s are | | `openbao-apps` | Every other namespace | `external-secrets-vault` in the `ExternalSecret`'s own namespace | `kv//` | | `openbao-claude-agents` | `claude-agents` | The same, in `claude-agents` | `kv/claude-*`, where `make claude-session-pat` writes each session's token | | Setting | Why | | --- | --- | | `openbao-apps` names no ServiceAccount namespace | ESO then requests the token in the namespace of the `ExternalSecret` or `PushSecret`, and the `external-secrets-apps` policy templates that namespace, as OpenBao read it from the token, into its paths. One role serves every workload, and a new one needs no change here | | `openbao` lists the platform namespaces by name, `openbao-apps` excludes the same list | Their prefixes hold the platform's credentials; a label would not do, because a workload may ship its own `Namespace` with any label. A platform component that reads a secret from a new namespace adds it to both | | Each workload ships an `external-secrets-vault` ServiceAccount | The role is bound to that name; `automountServiceAccountToken: false`, because only ESO needs a token for it | | Referent stores report `Valid` without logging in | ESO cannot know the namespace until an `ExternalSecret` names the store, so a missing role or ServiceAccount shows on the `ExternalSecret`, not the store | The ServiceAccount the platform store uses lives in `cluster-secret-store.yaml` too, because the stores only validate against a running, unsealed OpenBao. `make bao-init` writes the policies and roles on a fresh cluster by running `scripts/bao-policies.sh`; after a change to that script, `make bao-policies` rewrites them on a live one. It needs a root token, because the operator login cannot write policies: run it before `make bao-rotate-root`, or with a [generated root token](<#generating-a-root-token>). ## Usage ### Bootstrap 1. Initialise, unseal and configure the engine and ESO's auth. Safe to re-run: each configuration step is skipped if already in place, and an initialised cluster is left alone. ```bash make bao-init ``` > **Move the unseal keys before you do anything else** > > `make bao-init` writes the unseal keys to `output/credentials/openbao-init.json` and the [operator login](<#operator-credential>) to `openbao-operator.json` beside it (mode `0600`, gitignored) — a plaintext copy of the keys to every secret the cluster holds, next to the [etcd encryption key](). The root token in the first file is revoked before the command ends. Copy both into a password manager that does not need this cluster to be running, then delete the key file. Losing the keys means the data is unrecoverable: no support line, no recovery flow. 2. Write the values that belong to outside accounts — the prompts follow [Quickstart step 11](). The generated paths need nothing: their `PushSecret`s write them once the store validates. ```bash make bao-secrets ``` 3. Confirm the stores validate: ```bash kubectl get clustersecretstore ``` ### Authenticating locally ```bash kubectl -n openbao port-forward svc/openbao-active 8200:8200 & # a standby rejects requests export BAO_ADDR=http://127.0.0.1:8200 bao login -method=userpass username=operator # password from openbao-operator.json ``` ### Storing and reading a secret ```bash bao kv put kv/cert-manager/route53 access-key-id='AKIA...' secret-access-key='...' printf '%s' '{"access-key-id":"AKIA...","secret-access-key":"..."}' \ | bao kv put -mount=kv cert-manager/route53 - # what make bao-secrets does: JSON on stdin bao kv get kv/cert-manager/route53 ``` ### Rotating a credential A typed value is rotated with `make bao-secrets`, which asks per existing path whether to overwrite it; a generated one by deleting its path, after which the `PushSecret` writes a whole new set within its `refreshInterval`. 1. Rotate the path. For a typed value, answer `y` for that path; the others are left alone. `FORCE=1 make bao-secrets` overwrites every ordinary path without asking; `kneadtime/config` still wants a typed `OVERWRITE`, and without a terminal every existing path is left alone. ```bash make bao-secrets ``` For a generated path, delete it. Every key in it changes, with the consequences the reading component's page lists — for `authentik/config` that is every session, token and platform OIDC client at once: ```bash kubectl -n openbao exec openbao-0 -- bao kv metadata delete -mount=kv ``` 2. Push the new value into the `Secret` now rather than at the next `refreshInterval` — see [External Secrets](): ```bash kubectl annotate externalsecret -n force-sync=$(date +%s) --overwrite ``` 3. Restart whatever read the old value into an environment variable; a rotated `Secret` reaches a running pod only through a rollout. ## Health check ```bash for pod in $(kubectl -n openbao get pods -l app.kubernetes.io/name=openbao -o name); do kubectl -n openbao exec "$pod" -- bao status done kubectl get clustersecretstore kubectl get externalsecret -A ``` `bao status` should show `Initialized: true`, `Sealed: false`, and `HA Mode: active` on one pod with `standby` on the others; it exits `0` unsealed, `2` sealed and `1` unreachable. Every `ClusterSecretStore` should report `Ready=True`. `prometheusrule.yaml` mails `OpenBaoSealed` for each pod sealed past the rule's `for`, read from the `openbao-sealed` pod label because a sealed pod still reports Ready, and `OpenBaoNoActiveNode` when no unsealed active pod is scraped at all. ## Pitfalls > [!WARNING] > > **Sealed after every restart, and Ready does not mean unsealed** > > OpenBao seals itself on every pod restart — node reboot, chart bump, Kured — and no auto-unseal is configured, so it stays shut until someone with the key shares unseals it. The readiness and liveness probes answer healthy while sealed (`sealedcode=204`), deliberately, so a StatefulSet rollout does not stop at the first pod waiting for keys and a sealed pod is not restarted into another sealed pod; the cost is that only `bao status`, not `kubectl get pods`, can tell you OpenBao is usable. While sealed no `ExternalSecret` resolves, so cert-manager loses its Route53 credentials and nothing breaks until a certificate expires up to sixty days later, with no obvious link to the reboot. See [Limitations](). > [!WARNING] > > **A `PushSecret` cannot add a key to a path `make bao-secrets` wrote** > > Paths written before their `PushSecret` existed carry no `managed-by` metadata, so a new key on one fails with `secret not managed by external-secrets` and the `PushSecret` goes Degraded. The existing keys are unaffected. Mark the path as ESO's once, and the next reconcile adds the key: `bao kv metadata put -mount=kv -custom-metadata=managed-by=external-secrets `. > [!WARNING] > > **`#` and `!` on a command line** > > Inside double quotes an interactive bash expands `!` from history before the quotes are considered, and an unquoted `#` truncates the line. Both are silent and store a plausible credential that does not work. Use single quotes, or write the secret as JSON and pass `@file` — a missing file then fails loudly. ## Recovery ### Unsealing after a restart ```bash make bao-unseal ``` It feeds a threshold of shares from `output/credentials/openbao-init.json` to each sealed replica, waits for one that has not joined Raft yet, and is idempotent. With the key file deleted, which is the correct end state, unseal by hand: ```bash for pod in $(kubectl -n openbao get pods -l app.kubernetes.io/name=openbao -o name); do kubectl -n openbao exec "$pod" -- bao status >/dev/null 2>&1 case $? in 0) echo "$pod: already unsealed" ;; 2) for i in 1 2 3; do kubectl -n openbao exec -it "$pod" -- bao operator unseal done ;; *) echo "$pod: OpenBao did not answer, check the pod" ;; esac done ``` A replica that says `Vault is not initialized` has not joined the Raft cluster — one that predates the `retry_join` stanzas or started before `openbao-0` was initialised. Point it at the leader once; it takes unseal keys afterwards: ```bash kubectl -n openbao exec openbao-1 -- \ bao operator raft join http://openbao-0.openbao-internal:8200 ``` ### Generating a root token For what the operator policy does not cover — a policy, an auth method, a snapshot restore, or the operator login itself broken. Needs the operator login and a threshold of unseal shares; each step runs in the pod. 1. Log in as the operator; it prompts for the password from `openbao-operator.json`. ```bash kubectl -n openbao exec -it openbao-0 -- bao login -method=userpass username=operator ``` 2. Generate a one-time password and start the attempt with it; note both the OTP and the nonce. ```bash kubectl -n openbao exec openbao-0 -- bao operator generate-root -generate-otp kubectl -n openbao exec openbao-0 -- bao operator generate-root -init -otp= ``` 3. Enter one share per run until the threshold is met; the last prints `Encoded Token`. ```bash kubectl -n openbao exec -it openbao-0 -- bao operator generate-root -nonce= ``` 4. Decode it into the root token. ```bash kubectl -n openbao exec openbao-0 -- bao operator generate-root -decode= -otp= ``` 5. Do the one job with it, then revoke it and clear the pod's token cache. ```bash kubectl -n openbao exec -i openbao-0 -- bao login - # paste the root token, then Ctrl-D kubectl -n openbao exec openbao-0 -- bao token revoke -self kubectl -n openbao exec openbao-0 -- sh -c 'rm -f "$HOME/.bao-token"' ``` With the operator login gone too, no token can start an attempt: set `disable_unauthed_generate_root_endpoints = false` in the listener stanza in `application.yaml`, restart and unseal the pods, and drive the deprecated `sys/generate-root/*` API with the shares; revert the setting afterwards. To recreate the operator login with that root token, put it as `root_token` into a copy of `openbao-init.json`, move `openbao-operator.json` aside, and run `make bao-rotate-root`, which writes a new password and revokes the token. ### Backups Logged in as the operator through the `openbao-active` port-forward from [Authenticating locally](<#authenticating-locally>); its policy covers the snapshot, and a standby refuses it: ```bash bao operator raft snapshot save snapshot.bao ``` The snapshot holds all KV data and OpenBao's own configuration; restore with `bao operator raft snapshot restore`. Store it off-cluster — it is encrypted with a key that exists only in the unseal shares, so it is exactly as recoverable as your key custody. See [Backups & Recovery](). # Rook-Ceph Distributed block storage using [Rook]() as the Kubernetes operator for [Ceph](). It is the component that still has your data after a node dies, and the one with the steepest learning curve. Check `ceph status` more often than seems necessary and never ignore a `WARN`. ## At a glance | | | | --- | --- | | Namespace | `rook-ceph` | | Depends on | A raw `rook-osd` partition on every worker, written at install time | | If it is down | Every pod with a volume — `openbao` first, which turns a storage problem into a cluster problem | | Health check | `make storage-check` — it binds a real PVC, which `ceph status` alone does not prove | | UI | `rook.infra.k8s.wlkr.ch` | | Files | `payload/platform/rook-ceph-operator/`: the `rook-ceph` chart, the operator itself. `payload/platform/rook-ceph-cluster/`: the `rook-ceph-cluster` chart, whose `valuesObject` holds the `CephCluster`, the object store and the storage classes. `payload/platform/rook-ceph/`: the CSI driver and its RBAC, the dashboards and the UI route | ## Configuration ### Disks Each worker has a raw partition labelled `rook-osd`, created by Ignition as the **last** partition with whatever the root leaves, which Rook adds as an OSD; data is replicated three ways across them. Control planes get none, and the CephCluster's OSD placement excludes them: on one shared NVMe, etcd's fsyncs and BlueStore's writes stalled each other until OSDs hit their heartbeat timeout and aborted. Its contents are wherever Ceph last wrote them, so inserting a partition ahead of it shifts its offset and takes the OSD data with it: changing the partition table is a [reinstall](), not an edit. Ignition declares it with `format: none` and `wipe_filesystem: true`, because `ceph-volume` reads the BlueStore *signature*, not the partition table, and a reinstalled node that kept the old bytes would present an OSD from a cluster that no longer exists. Ceph declines any device that already carries a filesystem — the first thing to check when an OSD refuses to appear, see [No OSDs after reprovisioning](<#no-osds-after-reprovisioning>). ### Storage classes | StorageClass | Provides | Notes | | --- | --- | --- | | `rook-ceph-block` | RBD block volumes, cluster default | `ReadWriteOnce` only; `ReadWriteMany` needs CephFS, which is not enabled. Block PVs are deleted with their claim, and every Application prunes, so removing a PVC from Git removes the volume — see [Backups]() | | `ceph-bucket` | S3 buckets from the object store | Reclaim `Retain`, so deleting a claim cannot delete the bucket | ### Object storage A `CephObjectStore` provides an S3 endpoint at `http://rook-ceph-rgw-object-store.rook-ceph.svc`, so backups and log chunks live somewhere other than the block pool they protect. The infra Gateway publishes the same endpoint as `https://s3.infra.k8s.wlkr.ch` for [Velero](), whose S3 client only signs cleanly over TLS. More than one RGW instance (`gateway.instances`) keeps a node reboot from stalling a backup, which only holds while they sit on separate nodes: `gateway.placement` carries the spread constraint, because the `placement.all` block belongs to the `CephCluster` and never reaches this CR. It is a hard one: every gateway is daemon `a`, so on one node the exporter reports their counters under the same labels and Prometheus drops them, and the descheduler never evicts a `system-cluster-critical` pod to fix that. The data pool is erasure coded (`dataChunks` and `codingChunks`): less overhead than three replicas, safe at `failureDomain: host` only with more workers than chunks. ### Monitoring In `rook-ceph-cluster/application.yaml`'s `valuesObject`: | Setting | Why | | --- | --- | | `cephClusterSpec.monitoring.enabled` | Turns on the mgr `prometheus` module and lets the operator maintain a `ServiceMonitor`. Without it a degraded pool, a down OSD or a near-full cluster is invisible to Prometheus, and a full Ceph cluster stops accepting writes | | `monitoring.createPrometheusRules` | Ships Ceph's own alerting rules as a `PrometheusRule`. Its CRD comes from the `prometheus-operator-crds` Application; a sync before it lands fails and is retried — see [Monitoring]() | | `cephClusterSpec.cephConfig` `mgr/dashboard/*` | Sets the dashboard's Prometheus and Grafana links through the mon config store, which the operator reapplies on every reconcile, rather than a one-off `ceph dashboard set-*` Job | | `mon_cluster_log_level: info` | The Ceph default is `debug`, and the mons copy the cluster log to stderr, so Rook's probes and the mgr's pgmap ticks became most of the mon log volume in [Loki](). Warnings and errors are unaffected | Rook's Grafana dashboards are vendored in `grafana-dashboards.yaml` — see [Monitoring → Dashboards](). ### CSI driver From Rook v1.20 the RBD driver is deployed by the ceph-csi-operator subchart, which deploys nothing until an `OperatorConfig` and a `Driver` CR tell it what to run. Neither chart ships them, so they live in `csi-driver.yaml`; without them every PVC stays `Pending` on `ExternalProvisioning`. | Setting | Why | | --- | --- | | `csi-driver.yaml` in `rook-ceph`, with `SkipDryRunOnMissingResource` | The CRs need the ceph-csi-operator's CRDs, which arrive with the operator chart | | `OperatorConfig` pointing at the chart's image-set `ConfigMap` | A chart bump moves every sidecar and the cephcsi image together | | Driver name `rook-ceph.rbd.csi.ceph.com` | Derived from the namespace and must match the StorageClasses' `provisioner`. RBD is the only driver; CephFS is not enabled | | Two controller plugin replicas, `RollingUpdate` without surge, a PodDisruptionBudget | A drain of the provisioner's node otherwise stalls every provision, attach, resize and snapshot until the pod reschedules. The operator already keeps the replicas on distinct nodes with required anti-affinity and runs every sidecar with leader election, so the `Driver` CR carries no spread constraint, and no field for one | | `csi-rbac.yaml` | The ceph-csi-operator creates no ServiceAccounts, and the names are fixed by `CSI_SERVICE_ACCOUNT_PREFIX` (empty here): `rbd-ctrlplugin-sa` and `rbd-nodeplugin-sa`. Without them the Deployment and DaemonSet exist but create no pods | > [!WARNING] > > **`csi` values on the operator chart do nothing** > > Helm accepts values a chart no longer reads. Driver settings go in `csi-driver.yaml`. `csi-rbac.yaml` is the RBD half of the ceph-csi-operator's `deploy/multifile/csi-rbac.yaml` at the version the rook-ceph chart pulls in; Renovate does not see it. To refresh it: 1. Fetch and rename: ```bash curl -sSfL https://raw.githubusercontent.com/ceph/ceph-csi-operator/vX.Y.Z/deploy/multifile/csi-rbac.yaml \ | sed -e 's/ceph-csi-operator-system/rook-ceph/' -e 's/ceph-csi-operator-//' ``` 2. Keep only the rbd documents (CephFS, NFS and NVMe-oF are not enabled). 3. Re-indent the sequences for yamllint and prefix the cluster-scoped names. 4. Cut the nodeplugin ClusterRole back: drop `secrets`, `serviceaccounts` and `serviceaccounts/token`, and keep `get` on the `rook-csi-rbd-node` and `rook-csi-rbd-provisioner` Secrets in the `rbd-nodeplugin-r` Role. Its token sits on every node; upstream's grants serve only the encryption KMS providers, which nothing here uses, and minting any ServiceAccount's token is cluster-admin. ### cephx keys stay on aes Ceph 20 warns about the legacy `aes` cipher, and every key here is deliberately on it: the kernel client krbd does not implement `aes256k`, so CSI keys moved to it make every `rbd map` fail with `map failed: (22) Invalid argument` while Ceph reports the rotation a success. Two ciphers in one cluster trigger `AUTH_EMERGENCY_CIPHERS_SET` and buy no security. In `rook-ceph-cluster/application.yaml`'s `valuesObject`: | Setting | Why | | --- | --- | | `keyType: aes` | krbd cannot use anything else | | `mon_auth_allow_insecure_key: "true"` | Its default differs between mons; one that says `false` refuses an OSD's start-up key fetch, and an OSD whose on-disk key predates a rotation then fails with `Permission denied` | | `keyRotationPolicy: KeyGeneration` | Changing `keyType` alone never rotates existing keys while the policy is `Disabled` | | `keyGeneration` | Rotation only triggers on an increase, so a generation can never be lowered to undo one | | `healthCheck.muteHealthWarning` | The `AUTH_*` warnings would otherwise keep the Application `Degraded` while the data is healthy. Temporary: remove once RBD moves to the userspace `rbd-nbd` mounter and the keys can leave `aes` | ## Usage ### Requesting a volume Any workload can claim the default class: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-app-data namespace: my-app spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi ``` Add `storageClassName: rook-ceph-block` to say so explicitly. A Deployment with two replicas sharing one RWO PVC schedules one pod and leaves the other in `ContainerCreating` with a multi-attach error. ### Requesting a bucket Ask for an `ObjectBucketClaim`; Rook creates the bucket and writes a `ConfigMap` (`BUCKET_NAME`, `BUCKET_HOST`, `BUCKET_PORT`) and a `Secret` (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) sharing the claim's name: ```yaml apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: my-app-bucket namespace: my-app spec: generateBucketName: my-app storageClassName: ceph-bucket ``` ## Health check ### Is storage ready? ```bash make storage-check ``` Run it after the GitOps handover, before trusting anything that mounts a volume: a `CephCluster` reports `Ready` with no OSDs, and a `StorageClass` exists whether or not a CSI driver registered for it. The script walks the chain in the order it breaks and stops at the first missing link: | Check | What its absence means | | --- | --- | | `CephCluster` is `Ready` | the cluster Application has not synced yet | | an OSD pod is `Running` | Rook took no disk — see [No OSDs after reprovisioning](<#no-osds-after-reprovisioning>) | | `ceph health` is not `HEALTH_ERR` | Ceph itself is unwell; `HEALTH_WARN` is allowed through | | the CSI driver is registered | no `Driver` CR, so the ceph-csi-operator deployed nothing | | node plugin and provisioner are up | usually a ServiceAccount the DaemonSet cannot find | | a 1 GiB PVC binds and is cleaned up | the only check that proves the other five | The last one asks for a volume the way a workload would and waits `TIMEOUT` seconds (default in `scripts/storage-check.sh`); `NAMESPACE` and `CLASS` override where and what it asks for. ## Pitfalls > [!WARNING] > > **Ceph warnings show as Degraded** > > ArgoCD maps `HEALTH_WARN` to Degraded, so `rook-ceph-cluster` reports Degraded for the whole of a warning; nothing else waits on it — see [GitOps](). > [!WARNING] > > **Rook skips tainted nodes** > > The operator neither creates nor updates an OSD on a node with a taint `placement.all` does not tolerate, `PreferNoSchedule` included, and logs `placement settings do not match`. [Kured]() taints every node waiting for its reboot lock, so that taint is tolerated. ## Recovery ### No OSDs after reprovisioning A node whose `rook-osd` partition still carries a previous cluster's BlueStore signature gets no OSD: the `CephCluster` reports `Ready` with `HEALTH_WARN`, `kubectl -n rook-ceph get pods -l app=rook-ceph-osd` is empty, and the first PVC — `data-openbao-0` on a fresh bootstrap — stays `Pending`. > [!CAUTION] > > **The old cluster's data is already gone** > > Once the mons that held the cluster map left with the old control plane, those OSDs cannot be re-adopted by anything. Take a backup off the disks *before* a rebuild if you need one. 1. Confirm the cause: ```bash kubectl -n rook-ceph logs job/rook-ceph-osd-prepare- | tail -3 # skipping device "nvme0n1p10" because it contains a filesystem "ceph_bluestore" # skipping osd.3: "..." belonging to a different ceph cluster "..." ``` 2. Rebuild the node, steps 3 to 7 of [Rebuilding or repartitioning a node](): `make reinstall` only arms the PXE menu, and the [installer]() wipes the disk when the node network-boots. 3. If the node cannot be rebuilt now, wipe the partition by hand over SSH. `wipefs` removes the signature Rook skips on; the `dd` removes the BlueStore label behind it. Address it by label, as Ignition does, rather than by the device name in the log: ```bash sudo wipefs -a /dev/disk/by-partlabel/rook-osd sudo dd if=/dev/zero of=/dev/disk/by-partlabel/rook-osd bs=1M count=200 ``` 4. Restart the operator so it recreates the `osd-prepare` jobs: ```bash kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator ``` 5. Confirm an OSD per node and a binding PVC: ```bash make storage-check ``` # Security Policies Five cluster-wide controls: Pod Security Admission, default-deny network policy in both directions, a host firewall on the nodes, no API token on the `default` ServiceAccount, and admission policies on image references, namespace labels and the `authentik` namespace. Each is easy to turn on and hard to turn on safely, so the pattern is the same for all of them: measure first, enforce second, one namespace at a time. ## At a glance | | | | --- | --- | | Namespace | `kube-system` for the Application; it owns Namespace objects across the cluster | | Depends on | [Cilium]() to enforce the network policies | | If it is down | Nothing at the time; the labels and policies stay applied and only stop being corrected | | Health check | `kubectl get ns -L pod-security.kubernetes.io/enforce`, `kubectl get validatingadmissionpolicy` | | Pruning | Enabled, except for the Namespaces: each carries `argocd.argoproj.io/sync-options: Prune=false`, because pruning a Namespace deletes everything inside it, PVCs included. Anything else removed from Git is pruned, so the Application does not sit OutOfSync on leftovers | | Files | `payload/platform/security/`, the network policies one file per namespace under `network-policies/`, plus the CoreDNS PodDisruptionBudget — see [Components]() | ## Configuration ### Pod Security Admission A namespace without labels gets the cluster default from `ansible/templates/admission.yaml.j2`: `enforce` at `baseline`, `audit` and `warn` at `restricted`. It reaches a control-plane node only at provisioning; see [Node hardening](). It exempts `kube-system`, which Cilium needs before Argo CD can label anything, and the namespaces that need `privileged` but carry no labels. An exemption skips `audit` and `warn` as well, so the labelled namespaces below are not exempt. Each namespace in `pod-security.yaml` carries three labels: | Label | Set to | Effect | | --- | --- | --- | | `enforce` | The level the namespace demonstrably needs | Rejects pods that violate it | | `audit` | Stricter | Records violations in the [API server audit log]() | | `warn` | Stricter | Warns whoever applies the manifest | Enforcement is set to what already works, so nothing running breaks, while `warn` and `audit` show what a stricter level would catch; tightening later is an informed change. The level per namespace is in `pod-security.yaml`; the rule behind it: | Enforce | Which namespaces | Why not stricter | | --- | --- | --- | | `privileged` | Those with a component on the host: the CNI and control plane, Ceph's OSDs, node-exporter, Trivy's node-collector, the etcd jobs and Velero's node-agent, Alloy, Kured | Host networking, the host PID namespace, raw block devices or a hostPath under the kubelet, etcd or log directories, all of which `baseline` forbids. node-exporter has a namespace of its own (`namespaceOverride`) so the rest of the monitoring stack need not follow it | | `baseline` | Those with a container that starts as root or escalates: Grafana's `init-chown-data`, the runners and their job pods, and the operators not yet checked against `restricted` | Reduced when the chart offers a non-root path, one namespace at a time | | `restricted` | Everything else, and the namespaces that run no pods | — | `privileged` here means "not yet reduced", not "unexamined": `audit` and `warn` are still `baseline` or `restricted` on all of them. Namespace objects are owned by this Application so the labels stay declarative rather than drifting after `CreateNamespace=true`. `cilium-secrets` is the exception: the Cilium chart creates it, so its labels are `secretsNamespaceLabels` in `payload/platform/cilium/values.yaml`, and a second owner would fight the first. Workload namespaces label themselves in the workloads repository, and `namespace-privileged-allowed` keeps them off `privileged`. ### Network policies One file per namespace in `network-policies/`, holding that namespace's `CiliumNetworkPolicy` objects. A namespace gets two whole-namespace policies, `default-ingress` and `default-egress`, so every pod in it is default-deny in both directions, plus one policy per workload that needs more than the rest of the namespace does. Each policy's `description` is the record of who may connect and why; the rules are not repeated here. `cluster-wide.yaml` holds the `CiliumClusterwideNetworkPolicy` objects: the rules every pod gets, so a namespace file is the record of what its namespace may do *beyond* those. #### Why CiliumNetworkPolicy and not NetworkPolicy Two kinds of traffic have no pod identity a standard `NetworkPolicy` can name: Gateway traffic, which reaches a backend from the per-node Envoy, and kubelet health probes, which come from the node. A plain default-deny therefore kills ingress *and* probes, and the pods restart forever in a way that looks like an application fault. Cilium's `fromEntities` names them: `ingress` for Envoy-proxied traffic, `host` and `remote-node` for the kubelet. #### Cluster-wide rules A rule that is word-for-word the same in every namespace file is lifted into `cluster-wide.yaml`, selecting every endpoint that has a namespace and with `enableDefaultDeny` off in both directions. The selector is not `{}` because that also takes the Gateway's `reserved:ingress` endpoint and the health endpoint, which then carry a policy they have no use for. Default deny is off because a cluster-wide egress rule would otherwise put every pod in a namespace without policies into egress default-deny. DNS is the only rule lifted; the intra-namespace and node-probe rules would qualify but change what a namespace file says, so they stay. The second policy there, `init`, is for the seconds before the agent has resolved a new pod's labels. The pod then carries the reserved `init` identity, which no namespace policy selects, and the audit log shows its traffic denied all the same; a scan Job's first lookups and a starting pod's probes were the bulk of the audit log. It admits the node, DNS out and DNS in, nothing else. The third, `cnpg-backups`, selects every CloudNativePG pod by its `cnpg.io/cluster` label and admits only the object store, by the same two rules Velero's namespace has. Every database's backup sidecar needs exactly that, wherever the database runs, so one policy replaces a rule in each workload's file. #### What every policy contains | Rule | Why | | --- | --- | | Ingress from within the namespace | Replicas, sidecars and a workload's own database talk to each other on ports that change with the chart | | Ingress from `host` and `remote-node` | Kubelet probes come from the node, and so do the API server's webhook calls, the aggregation layer and `kubectl port-forward`; all of them carry the node's identity, on a control-plane node with the `kube-apiserver` label as well | | Ingress from `ingress` | Only where an HTTPRoute sends the Gateway's Envoy straight at the namespace; a namespace behind the Authentik outpost admits `authentik` instead. Kept at L4: the HTTPRoute is already the L7 filter for that traffic | | Ingress from `monitoring` on the scraped port, `GET /metrics` | The one port a ServiceMonitor or PodMonitor names, at L7, so the scraper can open nothing else | | Egress within the namespace, and to `kube-apiserver` where there is a Kubernetes client | The `kube-apiserver` entity is the endpoints behind `kubernetes.default`; no pod uses the kube-vip address. DNS is not repeated: the cluster-wide rule above sends every lookup through the proxy, which is what makes `toFQDNs` work and every lookup visible in Hubble | | Egress to external names as `toFQDNs` | A name reads as the dependency it is; an address does not. The Gateway's own addresses count as external: a pod calling `auth.k8s.wlkr.ch` is classified `world`, not `ingress` | | Egress to the backend pod as well, when the name is one the Gateway serves | The Gateway's Envoy checks the caller's egress policy a second time, against the backend pod it picked and that pod's port, and answers `403 Access denied` itself when no rule matches; audit mode never sees it. The OIDC clients of Authentik carry both rules | | HTTP rules on plaintext ports, ingress side only | A request crossing a namespace boundary is proxied once, by the receiving node. TLS and gRPC ports stay at L4; the proxy cannot read them. A machine caller gets the methods and paths a week of Hubble records showed; a browser UI behind the Authentik outpost stays at `http: [{}]`, its paths being the application's own. The path is matched with its query string: a rule anchored with `$` names the query as well, as OpenBao's `/v1/sys/metrics\?format=prometheus$` does | Host-networked pods (Cilium, kube-vip, the control plane, node-exporter, Tetragon, the CSI node plugins) have no endpoint of their own and cannot be selected; to everything else they are `host` or `remote-node`. `kube-system` therefore covers only its pod-networked workloads, each selected by label. Where a chart ships `NetworkPolicy` objects of its own (`argocd`) they are switched off in its values: Cilium unions the two kinds, so a chart rule that allows everything would silently override the default-deny. ### Host firewall Pod policies cannot select a host-networked port. A `CiliumClusterwideNetworkPolicy` with a `nodeSelector` can, and Cilium's [host firewall]() enforces it on the native device and `cilium_host`. What each port is for is in [Host Firewall Inventory 2026-10-30](). | Behaviour | Why it matters | | --- | --- | | Only packets addressed to a node IP are checked | The kube-vip address counts on the node holding it; Gateway addresses and NodePorts are translated first, so their policy is the backend pod's | | A node with no selecting policy is not denied anything on IPv4 | The switch can merge alone, and reverting the policy needs no second rollout | | No rule in a direction leaves that direction unenforced | A host policy with `ingress` only never blocks a node's own connections, and their replies pass through conntrack | | Host endpoint labels are filtered | `kubernetes.io/hostname` is not among them, so a node can only be chosen by a label such as `node-access` in the upstream example, or by the `control-plane` role label | | A pod reaching another node's IP is masqueraded to its own node | The target sees `remote-node`, so the firewall can tell the LAN from the cluster but not Prometheus from any other pod; the source pod's egress policy decides that | `network-policies/host-firewall.yaml` selects the nodes carrying `host-firewall.wlkr.ch/selected`. It denies the LAN everything but SSH, the API server, DHCP and ping, and admits the whole cluster to every port, since a narrower cluster rule could not name Prometheus and could lock out a node. ### Default ServiceAccount tokens A pod that names no `serviceAccountName` lands on `default`, which nothing configures, and gets an API token it has no use for. A workload reaching the API through `default` is already a misconfiguration, so `default-serviceaccounts.yaml` sets `automountServiceAccountToken: false` on it in every namespace with nothing to preserve, empty namespaces included. Left out: `kube-system`, where the control plane authenticates with client certificates and "very likely fine" is not the standard. `authentik-server` runs on `default` by chart default and is covered: Authentik's Kubernetes calls are worker tasks, which run under the `authentik` account. The same rule applies to a chart workload whose own ServiceAccount has no binding — Alertmanager, Cilium's Envoy and Argo CD's HAProxy switch the mount off in their values — unless the process needs the in-cluster config to start at all ([the External Secrets webhook]()) or the chart offers no switch (Rook's `rook-ceph-default` and `rook-ceph-rgw`). ### Agent RBAC `claude-agent-rbac.yaml` gives the in-cluster Claude Code agents two ClusterRoles for the `claude-reader` and `claude-writer` ServiceAccounts of the `claude-agents` namespace. Cluster-scoped RBAC cannot come from the `apps` project, and what an agent may touch is a platform decision. | Role | Bound | Grants | Leaves out | | --- | --- | --- | --- | | `claude-agent-read` | Cluster-wide, to both ServiceAccounts | `get`, `list`, `watch` on named core, workload and CRD kinds, `pods/log` included | Secrets, RBAC, webhooks, CRDs, `pods/exec`, Trivy's exposed-secret reports | | `claude-agent-write` | Nowhere by the platform: the workloads repository adds a RoleBinding in each namespace a writer session may change | Create, update, patch and delete on pods, workloads, Services, ConfigMaps, jobs, autoscalers and disruption budgets | The above, plus PersistentVolumeClaims (deleting one deletes data), NetworkPolicies and routes (they change who may reach what, which is a pull request) | - **Creating a pod reads every Secret in its namespace.** That is why the write role is bound per namespace, and the `apps` project cannot place a RoleBinding in the namespaces its [destinations]() exclude. - **`claude-writer-pod-limits`** in `admission-policies.yaml` refuses any pod or pod template from the writer that uses the host's network, PID or IPC namespace, a `hostPath` volume, a privileged container, or a ServiceAccount other than `default` or the one the workload already runs as. A namespace without a Pod Security level allows all of those, and another ServiceAccount's token is that account's RBAC. - **`pods/log` is cluster-wide**, so the reader sees whatever any component logs, platform namespaces included. Triage needs it, and a cluster-wide grant cannot leave out namespaces. Kinds are listed, never wildcarded, so a new CRD is invisible to the agents until someone adds it. Argo CD reverts a hand edit within minutes either way. ### Admission policies Trivy reports a bad image after it is running; `admission-policies.yaml` refuses or records it at admission, with the built-in `ValidatingAdmissionPolicy` rather than a policy engine. The image policies match Pods and every controller that produces them, so a bad Deployment is rejected at `kubectl apply` rather than failing quietly in its ReplicaSet. The other three close what RBAC and the `apps` project cannot express. | Policy | Action | Why | | --- | --- | --- | | `image-tag-pinned` | `Deny` | A tag other than `latest`, or a digest. Nothing running violates it, and an unpinned image is the one Renovate can neither track nor roll back | | `image-registry-allowed` | `Audit` and `Warn` | The registries the cluster pulls from today; a reference with no registry counts as `docker.io`. Recorded to the [audit log]() and shown to whoever applies, not enforced, until the record is clean | | `claude-writer-pod-limits` | `Deny` | Host access and borrowed ServiceAccounts from the agent writer, see [Agent RBAC](<#agent-rbac>) | | `namespace-privileged-allowed` | `Deny` | A Namespace may carry `enforce: privileged` only if it is in the policy's list, the `privileged` rows above. The `apps` project creates its own Namespaces and their labels, so without it a workload could opt out of Pod Security | | `authentik-foreign-objects` | `Deny` | In `authentik`, an object an Argo CD Application applied must come from `authentik` or `security`, or be the ConfigMap `authentik-blueprint-`. Argo CD cannot whitelist kinds per destination namespace, so this is what holds `apps` to its blueprint there; objects with no `argocd.argoproj.io/tracking-id` annotation, from controllers and `kubectl`, are not matched | All but one carry `failurePolicy: Ignore`: an expression that errors lets the request through rather than stopping every pod in the cluster. A real violation is still refused. `claude-writer-pod-limits` fails closed, since it only ever matches one identity. ## Usage ### Promoting the registry allowlist 1. Confirm nothing running would be refused, over the full Loki retention: ```text {job="kubernetes-audit"} |= "validation.policy.admission.k8s.io/validation_failure" |= "image-registry-allowed" ``` 2. Change `validationActions` on the binding in `admission-policies.yaml` to `["Deny"]`. ### Tightening a namespace 1. See what running pods a stricter level would reject, without changing anything: ```bash kubectl label --dry-run=server --overwrite ns cert-manager \ pod-security.kubernetes.io/enforce=restricted ``` 2. Catch what is not running right now — a nightly CronJob, say — from the `audit` label's records over the full Loki retention window, in Grafana: ```text {job="kubernetes-audit"} |= "pod-security.kubernetes.io/audit-violations" ``` 3. Raise `enforce` in `pod-security.yaml`. ### Rollout A new policy is enforced on arrival: `policyAuditMode` in `payload/platform/cilium/values.yaml` is off. Switched on, it makes every agent evaluate each verdict and log it with `AUDIT` instead of dropping, and those verdicts reach Loki the way drops do. The mode is cluster-wide, so while it is on the older policies are not enforced either, which is why a batch of new policies goes through it together. 1. Merge the values change; the agents roll on their own. `Enabled` on every node before anything else merges: ```bash kubectl -n kube-system rollout status ds/cilium kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg config get PolicyAuditMode ``` 2. Merge the policies and let them run for a week. What enforcement would have refused, by namespace pair: ```promql sum by (source_namespace, destination_namespace) (increase(hubble_policy_verdicts_total{action="audit"}[7d])) ``` and flow by flow, in Grafana: ```text {job="hubble", verdict="AUDIT"} | json | line_format "{{.flow_source_namespace}}/{{.flow_source_pod_name}} -> {{.flow_destination_namespace}}/{{.flow_destination_pod_name}} {{.flow_l4_TCP_destination_port}}{{.flow_l4_UDP_destination_port}}" ``` 3. Add a rule for every audited flow that is legitimate. Layer 7 rules are never audited: a request an HTTP rule does not match is answered `403` on the spot, which is why a new policy carries `http: [{}]` — proxied and recorded, nothing refused — until the week's requests have been read: ```text {job="hubble"} | json | flow_l7_http_method != "" | line_format "{{.flow_source_namespace}} -> {{.flow_destination_namespace}}:{{.flow_l4_TCP_destination_port}} {{.flow_l7_http_method}} {{.flow_l7_http_url}}" ``` 4. Tighten each `http: [{}]` to the methods and paths seen, set `policyAuditMode: false`, which rolls the agents again. The first week's reading is [Network Policy Audit 2026-09-27](), the reading before the flip [Network Policy Audit 2026-10-16](). From then on the verdict to watch is `DROPPED`, live or from Loki — see [Cilium]() — and the [`HubblePolicyDrops`]() alert mails each refused connection. If a single endpoint needs the same treatment later, audit mode can be set on it alone, until the agent restarts: ```bash kubectl -n kube-system exec ds/cilium -c cilium-agent -- \ cilium-dbg endpoint config PolicyAuditMode=Enabled ``` ### Host firewall rollout Audit is per host endpoint, a setting of the running agent: `policyAuditMode` would stop enforcing the pod policies, and upstream says the per-endpoint setting does not survive an agent restart, after which the node enforces. The audit week therefore runs through a node label, and every step before enforcement is reversible by removing that label. 1. Merge the switch; the agents roll, one node at a time. Every agent reports the firewall on the native device: ```bash kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep 'Host firewall' ``` 2. Reboot one worker with no policy selecting it and watch it return `Ready`, as in [Rebooting a node](). 3. Merge `payload/platform/cilium/host-firewall-audit.yaml`, a DaemonSet that puts each host endpoint back in audit mode within seconds of an agent restart, because any restart would otherwise start enforcing that node: a Renovate bump, a value change, Kured. Every node must answer `Enabled` before the policy merges: ```bash for p in $(kubectl -n kube-system get pods -l k8s-app=cilium -o name); do kubectl -n kube-system exec "$p" -c cilium-agent -- \ sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") | grep PolicyAuditMode' done ``` 4. Merge the policy; it selects nothing. Select one worker, with its console within reach, and confirm `Disabled (Audit)` on its `reserved:host` line: ```bash kubectl label node host-firewall.wlkr.ch/selected=true kubectl -n kube-system exec -c cilium-agent -- cilium-dbg endpoint list ``` 5. Reboot that worker. It must return `Ready`, and audit mode `Enabled` within a minute; if it does not, go to [Host firewall lockout](<#host-firewall-lockout>) and stop the rollout. 6. Select the other workers, then one control plane and reboot it, then the other two. Only then start the audit week. 7. Read a week of what enforcement would refuse. Pods and nodes are admitted, so anything here came from the LAN. Add a rule for what is legitimate: ```text {job="hubble", verdict="AUDIT"} | json dst_id="flow.destination.identity", src="flow.IP.source", dst="flow.IP.destination", tcp="flow.l4.TCP.destination_port", udp="flow.l4.UDP.destination_port" | dst_id="1" | line_format "{{.src}} -> {{.dst}} {{.tcp}}{{.udp}}" ``` 8. Enforce one worker: its label sends the DaemonSet away, and restarting its agent drops audit mode, rather than switching audit off at runtime. Check that `bpf policy get` lists separate identities and not one wildcard, and that SSH and the node's pods still work. Reboot it once more. ```bash kubectl label node host-firewall.wlkr.ch/enforce=true kubectl -n kube-system delete pod -l k8s-app=cilium --field-selector spec.nodeName= kubectl -n kube-system exec -c cilium-agent -- cilium-dbg bpf policy get ``` 9. Repeat for the workers, then the control planes one at a time. Then delete `payload/platform/cilium/host-firewall-audit.yaml` in a pull request. ## Health check ```bash kubectl get ns -L pod-security.kubernetes.io/enforce kubectl get cnp -A kubectl get ccnp kubectl get validatingadmissionpolicy ``` ## Pitfalls > [!NOTE] > > **Audit mode stops at the proxy** > > `policyAuditMode` is a datapath setting. A request the Gateway's Envoy or an HTTP rule refuses is answered `403 Access denied` on the spot, and Hubble records it as a forwarded response, not an audited verdict. Look for it with `hubble observe --http-status 403`; in Loki the denied request is a `DROPPED` flow of type `REQUEST` whose source carries the caller's namespace and the `ingress` identity. The agents' `argocd` calls failed this way, `server: envoy` and `grpc-message: Access denied`, while their policy named the host but not the `argocd-server` pod; see [Agent account](). > [!NOTE] > > **A name rule audits one SYN in twenty** > > A `toFQDNs` rule admits the addresses the DNS proxy has seen for the name, and the proxy releases the answer after at most 100ms whether or not every datapath has the new address. About one connection in twenty to a CDN name is recorded `AUDIT` with a `world` destination; under enforcement that `SYN` drops and the retransmit a second later goes through. A run of drops on one source port is different: the application connected to an address it cached past the DNS TTL. > [!NOTE] > > **The token change is not retroactive** > > The mount is decided at admission, so existing pods keep their token until recreated. That makes the change safe to roll out, and means a posture scan will not agree it is fixed until things restart. > [!CAUTION] > > **A reboot can isolate a node a host policy selects** > > [Upstream]() reports a selected node on this same combination, tunnel, kube-proxy replacement and WireGuard, dropping everything after a cold boot or a runtime audit-off, until an out-of-band fix. It is open and unreproduced here, and a power cut would hit every selected node at once. That is why the rollout selects one worker first and reboots it. ## Recovery ### Host firewall lockout Work down until the node answers. An agent that cannot reach the API server hears neither a policy change nor a label removal, so step 1 alone does not free a node that is already cut off. 1. API server answering: take the node out of the policy. Argo CD does not own node labels, so nothing puts it back. Use `--all` instead of a name for every node. ```bash kubectl label node host-firewall.wlkr.ch/selected- ``` 2. SSH answering: put the node in audit mode, which needs no API server and lasts until the agent restarts. ```bash ssh sudo bash -s <<'EOF' id=$(crictl ps -q --label io.kubernetes.container.name=cilium-agent) crictl exec "$id" sh -c 'cilium-dbg endpoint config $(cilium-dbg endpoint get -l reserved:host -o "jsonpath={\$[0].id}") PolicyAuditMode=Enabled' EOF ``` 3. Nothing answering, the API server up: nothing filters a node until its agent has started after boot, so stop the agent coming back. Run this from the workstation, then power-cycle the node, then do step 1 and start the kubelet again with `sudo systemctl unmask kubelet && sudo systemctl start kubelet`. ```bash until ssh -o ConnectTimeout=2 sudo systemctl mask --now kubelet; do sleep 1; done ``` 4. Every control plane cut off: the API server needs no CNI, so it comes up before the agent does. Power-cycle all three and take the label off in that window. ```bash until kubectl label node --all host-firewall.wlkr.ch/selected-; do sleep 2; done ``` 5. Still nothing: at the node's console, add `flatcar.autologin` to the boot entry in the GRUB menu, since `core` has no password, then run the inner command of step 2. Failing that, [rebuild the node](). 6. Then [unseal OpenBao]() and revert in Git: the policy first; the switch only if the firewall itself is the fault, since that revert rolls every agent again. # Tetragon [Tetragon]() watches processes at the kernel, with eBPF, on every node. Trivy says what an image *could* do; Tetragon records what a process *did*: every exec and exit in a pod, and, where a `TracingPolicy` says so, file opens and credential changes. It is the only thing here that sees a shell spawned in a running container. ## At a glance | | | | --- | --- | | Namespace | `kube-system` | | Depends on | Kernel BTF at `/sys/kernel/btf/vmlinux`, which Flatcar ships; [Logging]() for the events, [Monitoring]() for the alerts | | If it is down | No runtime record and no runtime alerts; nothing else notices | | Health check | `kubectl get tracingpolicies -o wide`; `TetragonPolicyNotLoaded` and `TetragonEventsLost` otherwise | | Dashboard | [Tetragon](): agent health, event rates, policy hits by binary, and the latest policy hits, shells and execs read from Loki | | Files | `payload/platform/tetragon/`: the chart, the policies, `prometheusrule.yaml` and a dashboard written for this cluster, since upstream ships none | ## Configuration Cilium's own chart, from `helm.cilium.io`; Tetragon does not depend on Cilium and runs privileged on the host network by the chart's default. | Setting | Why | | --- | --- | | Export to stdout | The chart runs a sidecar that tails Tetragon's JSON file to stdout, so [Alloy]() collects it as any container log: `{namespace="kube-system", container="export-stdout"}` | | `exportDenyList` | Exec and exit events from the host and from the platform's own `kube-system` pods, selected by label, stay out of the export; they are the bulk of the volume and the least interesting. Any other pod in `kube-system` is exported, because a stolen write role lands its privileged pod there. Policy hits still pass, which is the point of the `event_set` field | | `enableProcessCred` | Capabilities and `privileges_changed` on every exec, so a setuid binary or a file-capability escalation is visible without a policy | | Two `ServiceMonitor`s | `tetragon_policy_events_total` is what the alert reads; `tetragon_tracingpolicy_loaded` and the ring-buffer counters are its health | | Memory limit | 2.5x the measured working set of the busiest agent with the policy library loaded, per the [platform rule](); the dashboard's first row shows the current worst node against it | ### Policies `tracingpolicies.yaml` carries the cluster-scoped `TracingPolicy` objects, adapted from the [upstream policy library](). All post at most one event per minute per selector, and most watch containers only, since the host has the node journal; `sensitive-file-access` and the signature hook of `kernel-module-load` record the host too. Every hook carries a `message` and one of upstream's `observability.*` tags, so an event says what it is without a lookup here. | Policy | Hook | Fires on | Alerted | | --- | --- | --- | --- | | `sensitive-file-access` | `security_file_permission` | A read of a control-plane or etcd key, the encryption config, an admin kubeconfig, etcd's data or a kubelet key, matched by the end of the path so a hostPath mounted anywhere counts, unless the reader is a control-plane binary, the kubelet, `kubeadm` or `kube-vip` on the host network, or the path is under `/run/cilium/state/`, where Cilium's route reconciler keeps its own `.wal`; the alert leaves out `etcdctl` in the etcd backup and defrag Jobs, by namespace and Job name; a read of `/etc/shadow` on the host by anything but PAM's `/sbin/unix_chkpwd`, which every login and `sudo` runs (in a container any read is PAM in the image, so only writes count there); any write, host or container, to `/etc/passwd`, `/etc/shadow`, `/etc/group`, `/etc/sudoers`, `/etc/ld.so.preload`, or under `/etc/sudoers.d/`, `/etc/cron.d/` and `/root/.ssh/` | Yes: `TetragonSensitiveFileAccess` | | `process-creds-changed` | `commit_creds` | A container process that moved into another mount, pid, network or user namespace after its exec, except `runc`; the alert leaves out the `cilium` DaemonSet in `kube-system` and all of `rook-ceph`, whose init containers and OSD prepare Jobs `nsenter` into the host by design, and `/proc/self/exe` in the `arc-jobs` workflow pods, which is buildah re-executing itself into [its user namespace](). The hook runs on every execve and fork; the selector keeps the escape signal. Upstream's `matchCapabilityChanges` is not used: it matches a change in either direction, so every entrypoint that drops root fires it | Yes: `TetragonCredentialsEscalated` | | `exec-from-writable-path` | `security_bprm_check` | A container executing a file under `/tmp/`, `/var/tmp/`, `/dev/shm/`, `/run/`, `/var/run/`, `/shared/`, `/controller/` or `/plugins/`, except the two binaries an init container copies there and the scripts s6-overlay writes under `/run/s6*` and `/run/service/` at start | Yes: `TetragonExecFromWritablePath` | | `library-from-writable-path` | `security_mmap_file` | The same paths mapped with `PROT_EXEC`, which is how a dropped shared library loads, except the native libraries runtimes unpack under `/tmp` on every start: Alloy's Snowflake driver (`/tmp/gosnowflake-cgo*`) and zstd-jni in Dependency-Track (`/tmp/libzstd-jni-*`) | Yes: `TetragonLibraryFromWritablePath` | | `privileges-raise`, `privileges-raise-gid`, `privileges-raise-gid-argocd` | The `__sys_set*uid` and `__sys_set*gid` family | Any setuid or setgid to root inside a container. A record: root re-asserting root is routine (`logrotate`), and a setuid binary that actually raises privileges is `TetragonPrivilegedExec`. `runc`, which sets the ids on every container start, is dropped in the kernel. The redis-ha busybox calls setgid on every applet, so the gid hooks leave out pods labelled `app: redis-ha` in `argocd` only; one `podSelector` cannot say "this label in that namespace", hence two gid policies | No | | `bpf-program-load` | `bpf_check` | Any BPF program load from a container; the alert leaves out `cilium` and `kube-system` | Yes: `TetragonBpfProgramLoaded` | | `kernel-module-load` | `security_kernel_module_request`, `security_kernel_read_file`, `find_module_sections` | A module requested by name or read from a container, minus the `net-pf-2-proto-262` and `net-pf-10-proto-262` requests Go binaries and haproxy make when they probe MPTCP at start and the `net-pf-31` one Home Assistant makes when it looks for a Bluetooth adapter; the alert leaves out `rook-ceph`, whose CSI plugin loads `rbd` after a boot. The third hook records every load's signature check, host included: Flatcar signs its modules but does not enforce it (`CONFIG_MODULE_SIG` without `_FORCE`) | Yes: `TetragonKernelModuleLoaded`; `TetragonUnsignedKernelModule` from Loki | | `dns-outside-cluster` | `ip_output` | A port 53 packet from a container to anything but the kube-dns ClusterIP or the pod range, which is a hard-coded resolver in an image, a tunnel or exfiltration. The pod range is left out because it holds each node's Cilium router address, the pod's default gateway, which Home Assistant's dhcp integration asks for PTR records at start. The hook sees TCP and UDP alike, so musl's unconnected sends count too; the alert leaves out `kube-system`, where CoreDNS forwards upstream, and `cert-manager`, whose DNS-01 propagation check asks the zone's authoritative servers | Yes: `TetragonDnsOutsideCluster` | | `mount-in-container` | `security_sb_mount` | Any mount from a container except by `runc`, which mounts the rootfs, proc and sysfs on every start. Escapes start with a mount of the host's disk, cgroup or proc; the alert leaves out `rook-ceph`, `cilium` and `kube-system`, which mount by design | Yes: `TetragonMountInContainer` | | `ptrace-in-container` | `security_ptrace_access_check` | An attach-mode access to another process from a container: ptrace, `process_vm_writev`, `/proc//mem`. The read mode `ps` uses is left alone | Yes: `TetragonPtraceInContainer` | | `kill-unprivileged-user-namespace` | `create_user_ns` | A user namespace created without `CAP_SYS_ADMIN` in a container, killed, except in `arc-jobs`. The only policy in `enforce` mode | Yes: `TetragonUserNamespaceCreated` | | `unprivileged-user-namespace-arc-jobs` | `create_user_ns` | The same in `arc-jobs`, recorded: buildah builds the images there in a user namespace, see [GitHub runners]() | No; every image build would fire it | | `egress-outside-cluster` | `security_socket_connect` | An IPv4 `connect()`, TCP or UDP, from a container to anything outside the pod, service and site ranges | No; ACME, S3, the Trivy database and Home Assistant all do this routinely | The allow list in the first policy is the set of things that read key material in steady state, found by listing what fired. Add a binary there only after confirming it in the export; a person running `cat` on a node is exactly what the policy is for. A binary path is only trusted together with the host network namespace: any image can ship `/usr/local/bin/etcd`, but only a privileged namespace can run it with `hostNetwork`. The exemption stays in the kernel rather than on pod labels, because the `podSelector` pitfall below would miss a pod that reads a key the moment it starts; exemptions by namespace and workload go in the alert instead. The third policy is the runtime half of an image check: nothing in an image lives on those paths, so a binary executing from one was written after the container started. An emptyDir can be mounted anywhere, so the policy names the mounts this cluster puts binaries into rather than every emptyDir; `kubectl get pods -A -o json` lists the rest. A `readOnlyRootFilesystem` pod makes the check complete, since then a new binary has nowhere else to land. The offline half, every binary Tetragon saw executed diffed against the image's file list, needs no policy: query Loki by image and binary and resolve each path in `crane export - | tar t`. Four more detections from the same library need no policy, because the exec event already carries what they look for: `sudo`, a setuid or file-capability binary raising privileges, a fileless exec and a deleted binary. They are [Loki alerts](). Upstream's `sshd` policy is covered by `NodeSshLogin` from the node journal. ## Usage The dashboard is the summary; the queries below are what its Loki panels run. ```text # Everything a policy caught {namespace="kube-system", container="export-stdout"} | json | process_kprobe_policy_name!="" # Every exec in one namespace, with arguments {namespace="kube-system", container="export-stdout"} | json | process_exec_process_pod_namespace="home-assistant" | line_format "{{.process_exec_process_binary}} {{.process_exec_process_arguments}}" ``` The `tetra` CLI in the agent pod does the same live: ```bash kubectl -n kube-system exec ds/tetragon -c tetragon -- tetra getevents -o compact --namespaces home-assistant ``` ## Health check ```bash kubectl -n kube-system get ds tetragon kubectl get tracingpolicies -o wide # STATE should be enabled ``` ### Enforcement Every policy but one only records. `kill-unprivileged-user-namespace` carries `policy-mode: enforce` and a `Sigkill`, because nothing here creates user namespaces and most kernel escalation chains start with one. It stays scoped to containers: a sysctl would cover the host too, where systemd's `PrivateUsers=` needs them. The one exception is `arc-jobs`, where buildah needs one for every image build; its `podSelector` excludes that namespace, at the price of the first seconds of every container (see Pitfalls), and a monitor-only twin records the events there. The limits, both checked on the running nodes: `Override` and enforcers need `CONFIG_FUNCTION_ERROR_INJECTION`, which Flatcar's kernel does not set, and BPF LSM hooks need `bpf` in the `lsm=` boot argument, which Flatcar does not pass. That leaves `Sigkill`, which upstream warns does not stop an operation already in flight. A policy without `policy-mode` loads in monitor mode and its kill actions are elided, so an enforcing policy has to say so. ## Pitfalls > [!WARNING] > > **A `podSelector` misses a pod's first seconds** > > The policy filter learns pods from the API and maps their cgroups; a process that runs the moment a container starts is not yet selected, and the test pod's `unshare` went unrecorded by the filtered policy while the unfiltered one killed it. Put the hook an alert depends on in a policy without a `podSelector`, and do exclusions in PromQL. > [!WARNING] > > **A first hit is a new series, and `increase()` of one sample is zero** > > `tetragon_policy_events_total` has a pod label, so the first hit from a pod creates a counter series inside the alert window and `increase()` reports nothing until the second scrape moves it. Every policy alert therefore also matches a series absent one window earlier, with `unless ... offset`, and holds with `keep_firing_for`, because the agent drops a pod's series when the pod exits and a one-shot pod would otherwise flap. Copy that shape for a new alert, and test it with a pod that fires once, not with one that keeps firing. > [!WARNING] > > **A policy that does not load is silent** > > A `TracingPolicy` naming a kernel function this kernel does not export sits in state `error` and watches nothing. `TetragonPolicyNotLoaded` fires after its `for`; check `kubectl get tracingpolicies -o wide` after every Flatcar release that moves the kernel. # Trivy Operator Aqua Security's [Trivy Operator]() scans everything the cluster runs and writes every result back as a Kubernetes object. Reading those objects is [Vulnerability Triage](); this page is the operator itself. ## At a glance | | | | --- | --- | | Namespace | `trivy-system` | | Depends on | [Monitoring]() for the `ServiceMonitor` and the dashboard, [Rook-Ceph]() for the vulnerability database volume | | If it is down | Reports expire after `scannerReportTTL` and are not rebuilt; `TrivyContainerNotScanned` fires for every container | | Health check | `kubectl -n trivy-system get pods,jobs`, then the reconciliation below | | Dashboard | [Trivy Operator]() | | Files | `payload/platform/trivy-operator/`: the chart, `prometheusrule.yaml`, the `findings-history` and `report-prune` CronJobs, the `sbom-readers.yaml` grant and a dashboard written for this cluster. The namespace labels and the default-deny policy are in [Security Policies]() | | Scanner | Produces | Answers | | --- | --- | --- | | Vulnerability | `VulnerabilityReport` | Which CVEs are in the images we run | | Exposed secret | `ExposedSecretReport` | Which images have credentials baked in | | Config audit | `ConfigAuditReport` | Which workloads are misconfigured | | RBAC assessment | `RbacAssessmentReport` | Which roles grant too much | | Infra assessment | `InfraAssessmentReport` | Whether the nodes pass CIS | | Compliance | `ClusterComplianceReport` | CIS 1.23, NSA/CISA 1.0, PSS baseline and restricted | The `eks-` and `rke2-` compliance specs are for other distributions and are left out rather than reported as failing. ## Configuration The values in `application.yaml` that are not defaults: | Setting | Why | | --- | --- | | `scannerReportTTL`, `vulnerabilityScannerScanOnlyCurrentRevisions`, `configAuditScannerScanOnlyCurrentRevisions`, `scanJobTTL` | Reports live in etcd. A stale report is rebuilt, superseded ReplicaSets are not scanned, and finished jobs do not accumulate | | `builtInTrivyServer: true` | `ClientServer` mode: one `trivy-server` holds the vulnerability database on the PVC. Standalone mode downloads it in every scan job, which meets registry rate limits, and a rate-limited scan reports zero findings rather than an error | | `ignoreUnfixed: false`, every severity | A finding with no fix is usually the signal that the base image is wrong, and a finding the scanner could not grade is worth reading | | `ignorePolicy.authentik` | The image carries thousands of unfixed Debian findings and its reports exceed the etcd request limit. The Rego (v0 syntax, `object.get` because `FixedVersion` is absent when there is no fix) drops only findings that have no fix *and* are below HIGH | | `skipFiles` | Debian's placeholder key in every CloudNativePG image, which serves TLS from certificates of its own, would otherwise sit in every Postgres report. Trivy's Rego input for a secret has no path, so an ignore policy could only drop the `private-key` rule and every real key with it; this skips one exact path | | `additionalVulnerabilityReportFields` without `Description` and `Links` | A report is one API write, capped at 2 MiB; `Links` alone is over half the payload. An oversized report is rejected with nothing but an operator log line | | `operator.scanJobTimeout` and `trivy.timeout`, kept equal | The first is the Job's `activeDeadlineSeconds` and kills the pod whatever Trivy is doing; the second is Trivy's own flag and only fires if it is the shorter. Raising one alone does nothing | | `accessGlobalSecretsAndServiceAccount: false` | The default grants the operator read on every Secret in the cluster so scan jobs can borrow a workload's `imagePullSecrets`, and every image here is public. A private image would fail its scan and raise `TrivyContainerNotScanned`. The flag also drops `nodes/proxy`, without which node-collector silently skips the kubelet-configuration CIS checks, so `node-collector-rbac.yaml` grants that back | | `clusterSbomCacheEnabled: false` | With the cache, every workload after the first to use an image gets an `SbomReport` from the cached SBOM and no `VulnerabilityReport`. Upstream: [trivy-operator\#1668]() | | `configFile` with `scan.skip-version-check` and `scan.disable-telemetry` | The operator mounts it into every scan job as `trivy.yaml`. Trivy's version check and its usage telemetry share one request to `check.trivy.dev`, so switching off one still sends it; the server never makes the call, only the scan jobs | | `metricsVulnIdEnabled: false` | One series per CVE per container is a six-figure cardinality, and every label is already in the report | | `compliance.reportType: summary` | The collector skips reports in `all` format, so `trivy_compliance_info` disappears and the dashboard's failing-controls table empties. The `status` label is title-cased: `Fail`, `Pass` | | Control-plane tolerations on `nodeCollector` and `scanJobTolerations` | Without them the CIS infra assessment covers only the workers and reports a clean control plane it never looked at | | `enforce: privileged`, `audit`/`warn: restricted` on the namespace, in `payload/platform/security/pod-security.yaml` | node-collector hostPath-mounts `/var/lib/etcd`, `/var/lib/kubelet`, `/etc/kubernetes` and `/etc/cni/net.d`, which `baseline` forbids. Scan jobs themselves drop all capabilities and run read-only | | `resources` on the operator | Its cache holds every report in the cluster, and the `SbomReport`s are most of that. At the limit it is OOM-killed mid-write, which shows as reports that never appear rather than as an error | | `logDevMode: false` | Its `V(1)` lines are the only way to see decisions the operator makes silently, but it switches logging to console encoding. Turn it on to debug, then off | ### Alerting The rules in `prometheusrule.yaml`. Reports expire and are rebuilt every `scannerReportTTL`, so a "new" finding is one absent from the reports two days ago rather than from the previous scrape, and every rule waits longer than a rebuild. | Alert | Fires when | Why this shape | | --- | --- | --- | | `TrivyContainerNotScanned` | A running container has had no `VulnerabilityReport` for 2h | A rejected report is never written, so it looks identical to a clean one; only reconciling against what runs finds it | | `TrivyExposedSecretNew` | A secret finding, by rule and path, was in no report two days ago, whichever container carried it | An accepted finding never clears, so an absolute rule would fire forever; and the same finding in a new pod is not new | | `TrivyRbacCriticalNew` | A Role or ClusterRole gains a critical finding it did not have two days ago | Every critical ClusterRole is a chart's operator role or Kubernetes' own, see [Vulnerability Triage](); what matters is a new one | | `TrivyConfigAuditCritical` | Any workload fails a critical config-audit check for 2h | Nothing does today, so the rule is absolute | | `TrivyComplianceRegression` | A framework has more failing controls than a day ago, for 12h | Compliance is recomputed every six hours; two runs above yesterday is drift, not a rebuild | The two "new" rules also require the operator to have been reporting two days ago, so a fresh cluster does not page on every existing finding. `TrivyContainerNotScanned` compares every running container against the containers that have a `VulnerabilityReport` and names the difference: ```promql count by (namespace, container) ( kube_pod_container_info * on (namespace, pod) group_left () (max by (namespace, pod) (kube_pod_status_phase{phase="Running"} == 1)) ) unless on (namespace, container) count by (namespace, container) ( label_replace(trivy_image_info, "container", "$1", "container_name", "(.+)") ) ``` - **Keyed on `(namespace, container)`, not the digest**: containerd's platform digest and the registry's tag digest differ permanently for mirrored images. - **Restricted to `Running` pods**, or every completed Job counts as unscanned. - **A `for` of several full sweeps**, so the TTL churn does not page. The dashboard's stat and by-severity panels take `max by (image_digest)` before summing, and workload panels join ReplicaSets against `kube_replicaset_spec_replicas > 0`, so a finding counts once per image and a superseded revision drops out. The "Worst workloads" table stays per container on purpose: it answers where a finding runs. ### History Reports live for `scannerReportTTL` and Prometheus for its `retention`, so neither can answer whether a count went up over a quarter. The `findings-history` CronJob in `findings-history.yaml` lists every report kind once a day and prints one JSON line per namespace, and per compliance framework, to stdout; Alloy ships it like any container log and Loki keeps that one stream far longer than the rest (`retention_stream` in `loki/application.yaml`). The dashboard's last row reads it. Images are counted once per digest per namespace, the same rule the dashboard applies. ```text {namespace="trivy-system", container="findings-history"} | json | kind="vulnerabilities" | namespace="argocd" ``` ### SBOM export `sbomGenerationEnabled` makes the operator write one CycloneDX `SbomReport` per container next to the `VulnerabilityReport`. Nothing here reads them; [Dependency-Track]() in the workloads repository uploads every one of them nightly and keeps the history Trivy's 24h reports cannot. `sbom-readers.yaml` is the `ClusterRole` and binding that let its upload job list them, and the ReplicaSets, so it can skip the reports of superseded revisions: the `apps` project may not create cluster-scoped RBAC, and which workload reads the operator's findings is decided here. ### Report pruning `SbomReport`s and `ExposedSecretReport`s carry no TTL, unlike the `VulnerabilityReport`s: they are owned by the scanned ReplicaSet and live as long as it does, and Deployments keep ten superseded revisions scaled to zero. Those SBOMs were over half the reports' bytes in etcd, and each API server holds them decoded in its watch cache. The `report-prune` CronJob in `report-prune.yaml` deletes the two kinds for every ReplicaSet scaled to zero, daily before the Dependency-Track upload, which skips those reports anyway. The two `ScanOnlyCurrentRevisions` settings keep the operator from writing them again. Its log is one JSON line per kind with the count deleted. ## Usage The dashboard is a summary; the objects have the detail. ```bash # What has been scanned, worst first kubectl get vulnerabilityreports -A --sort-by='.report.summary.criticalCount' # The findings themselves, for one workload kubectl get vulnerabilityreports -n -o json \ | jq '.report.vulnerabilities[] | select(.severity=="CRITICAL") | {id: .vulnerabilityID, pkg: .resource, installed: .installedVersion, fixed: .fixedVersion}' # Misconfiguration, RBAC and node checks kubectl get configauditreports -A kubectl get rbacassessmentreports,clusterrbacassessmentreports -A kubectl get infraassessmentreports -A # Which compliance controls are failing kubectl get clustercompliancereports k8s-cis-1.23 -o json \ | jq '.status.summary, (.status.summaryReport.controlCheck[] | select(.totalFail > 0))' ``` ## Health check Reconcile running containers against scanned ones, keyed on `(namespace, container)` like the alert; the output is every running container with no `VulnerabilityReport`, and healthy is an empty output: ```bash comm -23 \ <(kubectl get pods -A --field-selector status.phase=Running -o json \ | jq -r '.items[] | .metadata.namespace as $ns | .spec.containers[] | "\($ns) \(.name)"' | sort -u) \ <(kubectl get vulnerabilityreports -A -o json \ | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.labels["trivy-operator.container.name"])"' | sort -u) ``` > [!CAUTION] > > **Zero is not the same as clean** > > A report that reads zero may be a clean image or a scan that failed: a rate-limited pull, a timeout, an image the operator could not resolve. Check the object exists and carries a real digest before believing a zero, and look at `kubectl -n trivy-system get jobs` when one never appears. This is where the `vulnerability-triage` skill in `.claude/skills/` starts. ## Pitfalls Four ways a workload ends up with no `VulnerabilityReport`, all silent from the outside: | Symptom | Cause | Where to look | | --- | --- | --- | | No report, no `SbomReport`, operator logs `ResourceExhausted ... larger than max` or `etcdserver: request is too large` | The report exceeds the API write ceiling | Operator log. Fix: the fields and ignore policies above | | No report, Job events show `DeadlineExceeded` | `scanJobTimeout` elapsed on a large image | `kubectl -n trivy-system get events --field-selector reason=DeadlineExceeded`, before `scanJobTTL` deletes the Job | | Some containers of a multi-container workload missing; job succeeded; log has `failed to analyze layer ... unexpected EOF` | A scan container died reading the image and the operator keeps only containers that exited 0. Hits workloads with several containers on one image (Rook, Cilium). Not the shared cache: concurrent scans with it pass | Scan job pod logs | | Reports missing at random, operator restart count climbing, last state `OOMKilled` | The report cache outgrew the memory limit | `kubectl -n trivy-system get pod -l app.kubernetes.io/name=trivy-operator -o jsonpath='{..lastState}'`. Fix: `resources` in `application.yaml` | | `SbomReport` exists, no `VulnerabilityReport` | SBOM cache reuse path ran `trivy sbom` and produced nothing | Confirm `clusterSbomCacheEnabled` is off | A completed scan job counts against `concurrentScanJobsLimit` until `scanJobTTL` deletes it, so a backlog drains in batches of the limit, the chart's default, once per TTL; that is the expected shape, not a stall. # Velero [Velero]() backs up Kubernetes objects and, through CSI snapshots and its data mover, the data in every PVC, nightly into a bucket in the Ceph object store. What it covers, what it does not, and the restore runbooks are in [Backups & Recovery](); this page is the chart tuning. ## At a glance | | | | --- | --- | | Namespace | `backup` | | Depends on | [Rook-Ceph]() for the bucket claims and RBD snapshots | | If it is down | Nightly backups stop; `VeleroBackupTooOld` fires once the last success is older than the rule's threshold | | Health check | `kubectl -n backup get backupstoragelocation` reads `Available` | | Files | `payload/platform/velero/`, `payload/platform/backup/`, `payload/platform/snapshot-controller/` | ## Configuration In `application.yaml`'s `valuesObject` unless a file is named: | Setting | Why | | --- | --- | | `checksumAlgorithm: ""` on the `BackupStorageLocation` | The AWS plugin's SDK sends a trailing checksum that Ceph RGW rejects with `XAmzContentSHA256Mismatch`; every upload fails without it | | `s3Url: https://s3.infra.k8s.wlkr.ch`, not the RGW Service | kopia's S3 client signs plain-HTTP uploads with the streaming SigV4 scheme, which Ceph RGW rejects as unsigned headers since [CVE-2026-54330](); over TLS it sends `UNSIGNED-PAYLOAD` and RGW accepts it. The infra Gateway terminates TLS with the Let's Encrypt wildcard, so no CA has to reach Velero or the movers | | `velero-plugin-for-aws` init container | The plugin minor must match the chart's Velero appVersion; the chart's commented example is one line behind | | `credentials.useSecret: false` with `extraEnvVars` | Keys come from the environment, fed by the `Secret` Rook writes for the `velero-bucket` claim; nothing is rendered into a file or Git | | `defaultSnapshotMoveData`, `deployNodeAgent`, `uploaderType: kopia` | A CSI snapshot is a Ceph object in the same cluster; the data mover (run by the node agent) streams it into the bucket and deletes the snapshot | | `volumeSnapshotLocation: []` | The chart's placeholder renders a `VolumeSnapshotLocation` the CRD rejects, failing every sync; Helm replaces lists, so the empty list removes it | | `VolumeSnapshotClass` in `payload/platform/backup/volumesnapshotclass.yaml` | The `velero.io/csi-volumesnapshot-class` label is how Velero finds the class; without it volumes are skipped silently. `deletionPolicy: Delete`, because the durable copy is the data mover's | | `snapshot-controller` Application | Neither kubeadm nor Rook installs the CSI snapshot controller or its CRDs, which must exist before the `VolumeSnapshotClass` applies, so it is its own Application. It applies the two kustomize directories [kubernetes-csi/external-snapshotter]() publishes at each tag, the owning project's only packaging, namespaced into `backup` with the hardening as a patch and the image pinned to the tag, because the Deployment upstream publishes at a tag still names the previous release's image, so Renovate moves the two together — see [Maintenance](); upstream ships no PDB for its two leader-elected replicas, so `pdb.yaml` adds one | | No `runAsNonRoot` on the `velero` container | Whether the plugin init container's copy into `/target` works non-root depends on the image's `USER`; confirm on a real backup first | | Node agent `containerSecurityContext` | Root and capabilities stay because kopia reads every pod volume; escalation is off and seccomp is on | | `prometheusRule` | Four rules: `VeleroBackupFailures` (critical) and `VeleroBackupPartialFailures` (warning) on a schedule's outcome, `VeleroBackupTooOld` (critical), the only one that catches a backup hanging `InProgress`, and `VeleroBackupInProgress` (info), which is [Kured]()'s reboot gate rather than a problem to mail | ### Repository password Kopia encrypts the volume data in every repository with one password, read from the Secret `velero-repo-credentials` (key `repository-password`) in Velero's namespace. When that Secret is missing Velero creates it with a default compiled into the binary, which protects nothing, so `payload/platform/backup/velero-repo-credentials.yaml` supplies it. | Setting | Why | | --- | --- | | A `Password` generator and `PushSecret` writing `velero/repository` | A [generated secret](): random, written once, and in the OpenBao snapshot that a restore on a fresh cluster needs anyway | | `creationPolicy: Owner` on the `ExternalSecret` | ESO takes over the Secret Velero created at start-up and overwrites the default | | `velero.io/exclude-from-backup` on the Secret | The resource backup lands in the same bucket as the repositories the password unlocks | | Secrets stay in the resource backup | The tarball under `backups/` is not encrypted, so every backed-up `Secret` sits in the bucket as JSON. Excluding them would break a namespace restore: CloudNativePG, the bucket claims, cert-manager and Argo CD generate Secrets that exist nowhere else. The `velero` bucket and its keys are therefore as sensitive as an unencrypted etcd | A repository keeps the password it was initialised with, and all repositories share one, so a changed value makes every existing repository unreachable; see [Re-initialising the repositories](<#re-initialising-the-repositories>). ## Usage ```bash velero -n backup backup create manual-$(date +%s) --include-namespaces my-app ``` ## Health check ```bash kubectl -n backup get backupstoragelocation # Available kubectl -n backup get backups.velero.io # Completed, not PartiallyFailed kubectl -n backup get volumesnapshotclass rook-ceph-block kubectl -n backup get backuprepositories.velero.io # Ready kubectl -n backup get externalsecret velero-repo-credentials # SecretSynced ``` ## Pitfalls > [!WARNING] > > **PartiallyFailed with no volume data** > > Velero found no labelled `VolumeSnapshotClass` and skipped every volume without an error. Check the class above exists and carries the label. > [!WARNING] > > **Every `BackupRepository` NotReady, `Access Denied` on `kopia.blobcfg`** > > Objects upload, volume data does not: the RGW log shows `PUT /velero/kopia/...` from `minio-go` with no authenticated user while the AWS SDK's PUTs succeed. The S3 URL is plain HTTP again; see the `s3Url` row above. > [!WARNING] > > **Every `BackupRepository` NotReady, `invalid repository password`** > > The password changed under existing repositories: OpenBao was rebuilt without its snapshot, `velero/repository` was deleted, or a backup ran on a fresh cluster before OpenBao was unsealed and initialised them with Velero's default. Volume data stops and backups end `PartiallyFailed`. Restore the old value into OpenBao if it exists anywhere; otherwise re-initialise as below. ## Recovery ### Re-initialising the repositories Discards every backup in the bucket, resource backups included: a `Backup` whose volume snapshots cannot be reached never finishes deleting, so the bucket is emptied as a whole and Velero drops the `Backup` objects it no longer finds there. 1. Confirm the Secret carries the value from OpenBao, not Velero's default. ```bash kubectl -n backup get externalsecret velero-repo-credentials # SecretSynced ``` 2. Delete the repository objects. ```bash kubectl -n backup delete backuprepositories.velero.io --all ``` 3. Empty the bucket with its own keys, from a machine that reaches the infra Gateway. ```bash export AWS_ACCESS_KEY_ID=$(kubectl -n backup get secret velero-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) export AWS_SECRET_ACCESS_KEY=$(kubectl -n backup get secret velero-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) export AWS_CONFIG_FILE=$(mktemp) # path-style addressing, without touching ~/.aws printf '[default]\ns3 =\n addressing_style = path\n' > "$AWS_CONFIG_FILE" aws --endpoint-url https://s3.infra.k8s.wlkr.ch --region eu-central-1 s3 rm s3://velero --recursive ``` 4. Restart Velero and the node agents, so no cached repository connection survives. ```bash kubectl -n backup rollout restart deployment/velero daemonset/node-agent ``` 5. Take a backup now rather than at the next schedule, and confirm it. ```bash velero -n backup backup create --from-schedule velero-daily kubectl -n backup get backups.velero.io # Completed kubectl -n backup get backuprepositories.velero.io # Ready ``` # Operations # Operations Everything here assumes a cluster that is already up; for first provisioning see the [Quickstart](). - **[Troubleshooting]()** --- The symptom-to-first-check table, and how to roll back a sync that made things worse. - **[Node Lifecycle]()** --- Rebooting, adding, replacing and rebuilding a node, including the two gates people skip: waiting for Ceph, and unsealing OpenBao afterwards. - **[Updates & Upgrades]()** --- The three mechanisms that update a node, and how to move a minor version deliberately. - **[Backups & Recovery]()** --- What is backed up, what is not, and the restore runbooks. - **[Control Plane VIP]()** --- How the API server address survives losing the node that answers it, and how to migrate a cluster built without it. - **[Encryption Key Rotation]()** --- Swapping the key that encrypts Secrets in etcd, with no API server restart and two rewrites of every Secret. - **[Vulnerability Triage]()** --- How to read the Trivy findings and when to file upstream; the dated logs are [Triage 2026-09-20]() and [Triage 2026-09-29](). ## Routine health check | Command | Healthy looks like | | --- | --- | | `kubectl get nodes` | Every node `Ready` | | `kubectl -n argocd get applications` | Every Application `Synced` *and* `Healthy` | | `kubectl get certificate -A` | Every certificate `READY=True` | | `ssh sudo kubeadm certs check-expiration` | More than thirty days left on every line; less and the [yearly renewal]() is due | | `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status` | `HEALTH_OK`, and an OSD count matching your node count | | `kubectl -n openbao exec openbao-0 -- bao status` | `Sealed: false`; the one failure that looks like health, so run it after anything restarted | The Rook toolbox pod is enabled, so `ceph status`, `ceph osd tree` and `ceph health detail` need nothing installed. Ceph metrics are scraped and Ceph's own alerting rules are loaded, so a degraded pool reaches Prometheus without anyone running `ceph status`; run it anyway, because it is the fastest way to see *why* — see [Rook-Ceph → Monitoring](). # Backups & Recovery Every mechanism below is described by what it can get back, and by what it cannot. ## Current state | Data | Backed up | How | | --- | --- | --- | | Kubernetes objects | Nightly | [Velero](<#velero>) to the Ceph object store | | Ceph RBD volumes (PVCs) | Nightly | Velero CSI snapshot, moved into the object store | | PostgreSQL databases | Continuous WAL, nightly base backup | [CloudNativePG's Barman Cloud plugin](), one bucket per database; restores to a point in time | | Grafana dashboards | Nightly | Its PVC is covered by the above | | etcd (raw) | Nightly | [CronJob](<#etcd>) to the object store | | OpenBao secrets | Manual | [Raft snapshot](<#openbao>) | | OpenBao unseal keys | Manual, off-cluster | Printed once at `bao operator init` | | Prometheus metrics | **No** | Its PVC carries `velero.io/exclude-from-backup`; metrics live for the `retention` in `payload/platform/monitoring/application.yaml`, then are gone | | Trivy reports and vulnerability database | **No** | `trivy-system` is excluded; the operator rescans and the server downloads the database again | | Everything in `payload/` | Yes | It is in Git | ## What is not covered **Every automated backup lands in the same cluster's Ceph.** That protects against operator error (a deleted PVC, a bad `prune`, a corrupted database) and not against losing the cluster, because the backups go with it. Off-site replication is the missing piece: RGW supports bucket replication and Velero a second `BackupStorageLocation`, and neither is configured. Until then, disaster recovery is **Git, the OpenBao unseal keys, and a copy of the latest OpenBao snapshot and of the two buckets kept off-cluster** ([Copying a bucket off the cluster](<#copying-a-bucket-off-the-cluster>)); the nightly backups protect against mistakes. Velero's `PrometheusRule` reports a backup that failed (`VeleroBackupFailures`, `VeleroBackupPartialFailures`); only `VeleroBackupTooOld` notices one that silently stopped, since a backup hung `InProgress` never fails ([Velero]()). All of them only reach whoever Alertmanager is configured to tell — see [Monitoring](). ## Velero [Velero]() backs up Kubernetes objects and volume data nightly. Chart values and why each is set are in [Velero](). | Property | Value | | --- | --- | | Schedule | The `schedules` block in `payload/platform/velero/application.yaml`: nightly, an hour after etcd so the two never contend for the object store, with a TTL of two weeks so a bad backup is noticed before the last good one expires | | Scope | All namespaces except `kube-system` (rebuilt from Git, and large) and `trivy-system` (rescanned), minus `events` (they expire anyway) and any PVC labelled `velero.io/exclude-from-backup`, which is only Prometheus's: the chart's `volumeClaimTemplate.metadata.labels` carries it | | Destination | S3 bucket `velero` in the [Ceph object store]() | | Volume data | CSI snapshot, streamed into the bucket by the data mover (Kopia), snapshot deleted; the durable copy is the one in the bucket | | Encryption | Volume data with the repository password generated into OpenBao at `velero/repository`, which a restore needs; resource backups, Secrets included, are plain JSON — see [Repository password]() | The `velero` CLI defaults to the `velero` namespace; this install is in `backup`, so pass `-n backup` every time. ```bash kubectl -n backup get backups.velero.io kubectl -n backup get backupstoragelocation # should be Available velero -n backup backup create manual-$(date +%s) --include-namespaces my-app ``` Backups in `PartiallyFailed` with no volume data mean Velero found no `VolumeSnapshotClass` labelled `velero.io/csi-volumesnapshot-class: "true"` and skipped the volumes silently — check `payload/platform/backup/volumesnapshotclass.yaml` is applied. ## etcd Velero restores *through* the API server, which is no use with no API server left (lost quorum, a corrupted data directory). For that a CronJob in `payload/platform/backup/etcd-backup.yaml` snapshots etcd nightly, an hour before Velero runs. | Property | Value | | --- | --- | | Schedule | `payload/platform/backup/etcd-backup.yaml`: nightly, before Velero | | Where it runs | Any control-plane node, on the host network: etcd listens on `127.0.0.1` and its client certs are on the node. It mounts the CA certificate and the healthcheck client pair, never the directory, which also holds the CA and server keys | | Verification | `etcdutl snapshot status` before upload, so a truncated snapshot fails the job instead of replacing a good backup (`etcdctl snapshot status` was removed in etcd 3.6) | | Destination | S3 bucket `etcd-backup`, separate from Velero's because it is restored by different means; the job keeps the newest snapshots and deletes the rest, the count is in the manifest | Two settings there are easy to undo by accident: the DNS policy for a host-network pod (the node's `resolv.conf` cannot resolve the RGW Service), and the AWS CLI config file selecting path-style addressing (no environment variable sets it). ```bash kubectl -n backup get cronjob etcd-backup kubectl -n backup logs job/ -c upload ``` ### Defragmentation The API server compacts etcd's history every five minutes, but compaction only frees pages inside the database file; the file itself never shrinks until a member is defragmented, and once less than half of it is in use `etcdDatabaseHighFragmentationRatio` fires. A second CronJob in `payload/platform/backup/etcd-defrag.yaml` runs `etcdctl defrag --cluster` nightly from the same kind of host-network pod as the backup, one member after another, and prints `endpoint status` before and after so the log shows the size reclaimed. | Property | Why | | --- | --- | | Nightly | This cluster's churn crosses the alert's ratio about a day and a half after a defrag, so the alert firing means a run failed or did not help, not that the next one is due | | One member at a time | A member blocks for seconds while it rewrites its file; the other two keep quorum | | Fails when a member is unreachable | `endpoint status --cluster` exits non-zero, so a cluster already short of a member is not defragmented | ```bash kubectl -n backup get cronjob etcd-defrag kubectl -n backup create job --from=cronjob/etcd-defrag etcd-defrag-manual kubectl -n backup logs job/etcd-defrag-manual ``` ### Taking one by hand The snapshot lands in the static pod's `/var/lib/etcd`, which is the node's; `kubectl cp` cannot fetch it, because the etcd image is distroless and has no `tar`. ```bash kubectl -n kube-system exec -it etcd- -- etcdctl \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ snapshot save /var/lib/etcd/snapshot.db ssh sudo cat /var/lib/etcd/snapshot.db > ./etcd-snapshot.db ssh sudo rm /var/lib/etcd/snapshot.db ``` Keep the copy off the cluster, like the nightly ones below. ### Copying a bucket off the cluster Rook writes each bucket's keys into a `Secret` named after its claim (`velero-bucket`, `etcd-backup-bucket` in `backup`) and the RGW answers on the infra Gateway; see [Requesting a bucket](). 1. Export the bucket's own keys and the path-style config RGW needs, as in [Re-initialising the repositories](). ```bash export AWS_ACCESS_KEY_ID=$(kubectl -n backup get secret -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) export AWS_SECRET_ACCESS_KEY=$(kubectl -n backup get secret -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) export AWS_CONFIG_FILE=$(mktemp) printf '[default]\ns3 =\n addressing_style = path\n' > "$AWS_CONFIG_FILE" ``` 2. Mirror the bucket to a disk that is not in the cluster; `sync` is incremental, so a repeat copies only what the last night added. ```bash aws --endpoint-url https://s3.infra.k8s.wlkr.ch --region eu-central-1 s3 sync s3:// ./ ``` The copy is as plain as the bucket: Velero's resource backups are JSON with Secrets in them, so store it with the unseal keys, not beside the Git clone. ## OpenBao ```bash kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot save /tmp/snapshot.bao kubectl -n openbao cp openbao-0:/tmp/snapshot.bao ./openbao-snapshot.bao ``` The snapshot holds all KV data plus policies, roles and mounts. It does **not** hold the unseal keys and is useless without them — see [OpenBao](). ## Ceph volumes PVCs are covered by [Velero](<#velero>). Ceph's own replication is not a backup: it spreads each block across OSDs, which survives a disk or node failing and nothing else. Almost every Application runs with `prune: true` (`kube-vip` and `security` are the deliberate exceptions), so removing a `PersistentVolumeClaim` from Git deletes the volume. ## Restoring The commands are the upstream-documented ones for the versions pinned in `payload/platform/`; the etcd sequence has not been exercised on this cluster. ### Velero restore 1. Find the backup. ```bash velero -n backup backup get ``` 2. Restore from it, scoped to what was lost; without `--include-namespaces` everything in the backup is restored, and existing objects are skipped, not overwritten. ```bash velero -n backup restore create --from-backup velero-daily- \ --include-namespaces my-app ``` 3. Watch it finish and read the warnings. ```bash velero -n backup restore describe velero -n backup restore logs ``` ### OpenBao snapshot restore Needs an initialised, unsealed cluster and a root token from [Generating a root token](). Afterwards the snapshot's own operator login applies, not the new cluster's. `-force` is what lets a snapshot from a *different* cluster (a fresh `bao operator init`) load; afterwards the pods seal and want the snapshot's keys, not the new cluster's. 1. Copy the snapshot in. ```bash kubectl -n openbao cp ./openbao-snapshot.bao openbao-0:/tmp/snapshot.bao ``` 2. Restore it. ```bash kubectl -n openbao exec -it openbao-0 -- \ env BAO_TOKEN= bao operator raft snapshot restore -force /tmp/snapshot.bao ``` 3. Unseal every replica with the snapshot's key shares and confirm. The script reads `output/credentials/openbao-init.json`, which now holds the new cluster's keys, so point it at the old file with `KEYFILE`. ```bash KEYFILE= make bao-unseal kubectl -n openbao exec openbao-0 -- bao status # Sealed: false ``` 4. Put the old key and operator files in place of the new ones, which unseal nothing any more; every later `make bao-*` reads them. ```bash mv output/credentials/openbao-init.json mv output/credentials/openbao-operator.json ``` ### etcd restore Every etcd member restores from the same snapshot, then all three static pods come back together. Flatcar ships no `etcdutl`, so run it from the etcd image the CronJob uses. Upstream: [Restoring an etcd cluster](). 1. Copy the snapshot, from the off-cluster copy of the `etcd-backup` bucket ([Copying a bucket off the cluster](<#copying-a-bucket-off-the-cluster>)), to every control-plane node as `/var/lib/etcd-restore/snapshot.db`. 2. On every control-plane node, stop the API server and etcd by moving their static pod manifests out, and wait until neither container is listed. ```bash sudo mkdir -p /root/manifests-stopped sudo mv /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml /root/manifests-stopped/ sudo crictl ps --name 'etcd|kube-apiserver' # empty ``` 3. On every control-plane node, pull the etcd image the CronJob uses (the tag is in `payload/platform/backup/etcd-backup.yaml`). Only the node that last ran the CronJob has it; `ctr run` does not pull. ```bash sudo ctr -n k8s.io image pull registry.k8s.io/etcd: ``` 4. On every control-plane node, restore into a new data directory with the `--name`, `--initial-cluster` and `--initial-advertise-peer-urls` values from that node's `/etc/kubernetes/manifests/etcd.yaml`, and one `--initial-cluster-token` that is the same on every node and differs from the old cluster's, so a member that missed the restore cannot rejoin the new one. ```bash sudo ctr -n k8s.io run --rm \ --mount type=bind,src=/var/lib/etcd-restore,dst=/restore,options=rbind:rw \ registry.k8s.io/etcd: etcd-restore \ etcdutl snapshot restore /restore/snapshot.db --data-dir /restore/etcd \ --name --initial-cluster \ --initial-cluster-token \ --initial-advertise-peer-urls https://:2380 ``` 5. Swap the data directory in, keeping the old one and its ownership. ```bash sudo mv /var/lib/etcd /var/lib/etcd.bak sudo mv /var/lib/etcd-restore/etcd /var/lib/etcd sudo chown -R --reference=/var/lib/etcd.bak /var/lib/etcd ``` 6. Put the manifests back on every node, then confirm. ```bash sudo mv /root/manifests-stopped/*.yaml /etc/kubernetes/manifests/ kubectl get nodes kubectl -n argocd get applications # ArgoCD reconverges from Git ``` ### Rebuild from scratch 1. Have the Git repository, the OpenBao unseal keys and operator login, an OpenBao snapshot and the off-cluster copy of the `velero` bucket; an etcd snapshot is optional and skips re-issuing certificates. 2. Move the old `openbao-init.json` and `openbao-operator.json` out of `output/credentials/`: `make bao-init` refuses while either exists, because keys from another cluster unseal nothing. 3. Provision the cluster per the [Quickstart]() from step 1. 4. When every `ExternalSecret` is Degraded, `make bao-init`, then either restore the OpenBao snapshot as above, which puts the old files back, or re-enter the typed secrets with `make bao-secrets` and let the `PushSecret`s generate the rest anew — which loses whatever was encrypted with the old keys, Velero's volume backups among them. 5. Once the platform is Synced and Healthy, restore workloads with a Velero restore from the copied-off `velero` bucket. # Control Plane VIP The API server is reached through a virtual IP held by [kube-vip](), not through one named node: whichever control-plane node wins the leader election answers on `control_plane_vip` from `ansible/inventory.yaml`, and another takes over if it goes away. The address is fixed at `kubeadm init` — see [Known Limitations](); a cluster built before the VIP is [migrated in place](<#migrating-a-cluster-built-without-a-vip>). ## How it is wired | Piece | Where | | --- | --- | | VIP address and interface | `control_plane_vip`, `control_plane_vip_interface` in `ansible/inventory.yaml` | | Bootstrap static pod | Written by Ignition to `/etc/kubernetes/manifests/kube-vip.yaml` on control-plane nodes, at `kube_vip_version`, because it must run before `kubeadm init` writes the endpoint into the certificates | | Running kube-vip | DaemonSet in `payload/platform/kube-vip/`, synced by ArgoCD; its image is tracked by Renovate | | Cluster endpoint | `controlPlaneEndpoint` in the generated kubeadm config, and every `kubeadm join`, from `control_plane_endpoint_host` in `ansible/inventory.yaml` | | Cilium's API address | `k8sServiceHost` in `payload/platform/cilium/values.yaml` | kube-vip runs in ARP mode with leader election, advertising a `/32`, with `svc_enable: "false"`: LoadBalancer services belong to Cilium's L2 announcements, and two components answering ARP for the same range produces intermittent, host-dependent failures. ## Design The static pod is only the bootstrap: once ArgoCD syncs `payload/platform/kube-vip/`, a DaemonSet takes over, so upgrading kube-vip is a merged PR rather than a file rewritten on each node. kube-vip publishes a chart; it is not used because the handover below needs the `adopt` init container, which the chart has no hook for. A DaemonSet can hold the VIP here because the control-plane kubelets use their own node's API server (`server:` in `/etc/kubernetes/kubelet.conf`) and so does kube-vip (`KUBERNETES_SERVICE_HOST` is the host IP, which is in the certificate SANs), so both start after a full power loss without the VIP or Cilium. `kube_vip_version` in the inventory is therefore the bootstrap version only, and the VIP settings in the inventory and `daemonset.yaml` must be kept in step. The first sync replaces all three static pods at once and drops the VIP for a few seconds; later updates roll one node at a time. kube-vip's ServiceAccount may only get and update its own Lease, and the Application does not prune and its objects carry `Delete=false`, because removing them takes the API away from the workers and from every kubeconfig. ### The handover | Detail | Why | | --- | --- | | `pull` init container | Fetches the kube-vip image first, so the gap between old and new process has no registry round trip. Keep its image in step with the main container | | `adopt` init container | Deletes the static manifest and waits for the static kube-vip to release `:2112`; both use the node name as leader identity and the same `plndr-cp-lock` Lease, so they must never run side by side | | `/etc/kubernetes/manifests` mounted as a directory | A hostPath mounted as a single file is a bind mount and cannot be removed | | 120 s timeout in `adopt` | The kubelet rescans manifests every 20 s; giving up leaves a visible stuck `Init` rather than a second kube-vip on the port | | `maxUnavailable: 1`, no surge | An old and a new pod would fight over `:2112` and the lease | | Tolerates every taint | A NotReady or pressured control-plane node is when the VIP has to move | | Capabilities identical to the static pod | The first sync replaces all three at once; untested hardening takes the VIP down everywhere | | Liveness probe on `/metrics` | Every kube-vip serves it, leader or not, so a hung process holding the Lease is restarted instead of keeping the VIP dead until another node's renew fails too; the long failure window keeps a slow API server from recycling a healthy pod | ### The `adopt` trade-off `adopt` and its writable hostPath stay after the handover, although on an adopted node they do nothing and every pod start still runs a container that can delete the control plane's static pod manifests. A reinstalled control-plane node gets the static manifest back from Ignition and needs adoption again, and forgetting to re-add it for that one reinstall leaves two kube-vips contending for the API VIP; the exposure is limited to the pinned image. > [!WARNING] > > **Tetragon holds `:2112` on the control planes** > > Tetragon serves its metrics on the same port, on the host network by its chart's default, so `adopt` waits for a port that is never released and gives up after its timeout on a reinstalled control plane, and the liveness probe is answered by Tetragon whether kube-vip is alive or not, as recorded in the [host firewall inventory](). Not fixed yet; move one of the two ports before the next control-plane reinstall. > [!CAUTION] > > **If the DaemonSet pods do not come up** > > The static manifests are already gone, so nothing holds the VIP, and every kubeconfig and Cilium's `k8sServiceHost` point at it. Running pods keep their networking, but no Cilium agent sees the API server until the VIP is back. Point `kubectl` at a control-plane node's `ansible_host` from `ansible/inventory.yaml` with `--server https://
:6443` (the node addresses are in the certificate SANs), write `/etc/kubernetes/manifests/kube-vip.yaml` back onto one control-plane node from `ansible/templates/butane_node_config.yaml.j2` so that the VIP answers again, then read `kubectl -n kube-system logs ds/kube-vip -c kube-vip` and fix the DaemonSet in Git. The static pod holds the VIP only while the broken DaemonSet pod merely restarts; if that pod is deleted or recreated, `adopt` removes the file again. > [!NOTE] > > Since Kubernetes 1.29, `admin.conf` is not usable until `kubeadm init` finishes, so the bootstrap unit points kube-vip's `hostPath` at `super-admin.conf` during init and moves it back afterwards. Inside the container the file stays at `/etc/kubernetes/admin.conf`. ## Endpoint name The next cluster build uses a DNS name as `controlPlaneEndpoint`, so moving the VIP later no longer means rewriting every node's kubeconfig; `certSANs` still needs regenerating for the new address. The running cluster keeps the bare VIP: there is no migration, because its certificates and stored configuration do not carry a name. | Piece | With a name | Why | | --- | --- | --- | | `/etc/hosts` from Ignition, every node | Resolves the name to the VIP | The API endpoint must not depend on LAN DNS | | `apiServer.certSANs` | Holds the VIP | Cilium's `k8sServiceHost` and `make kubeconfig` keep using the VIP | | `make kubeconfig` | Writes the VIP as `server` | Only the nodes resolve the name | | `make config`, `make reinstall` | Refuse a name the answering cluster does not publish | A node rendered with it fails certificate verification on join, after its disk is wiped | To build with a name: 1. Set `control_plane_endpoint_host` in `ansible/inventory.yaml` to the name, before the first `make config` of the build, and keep it there. ```yaml control_plane_endpoint_host: k8s-api. ``` 2. Continue with the [Quickstart](). Everything else follows from that variable. Check that nodes and the cluster agree once the first control plane is up. ```bash kubectl -n kube-public get configmap cluster-info -o jsonpath='{.data.kubeconfig}' | grep server ``` ## Choosing the address A free address on the nodes' subnet, outside any DHCP range, and distinct from the Cilium LoadBalancer pools. Nothing validates this: a collision shows up as an API server unreachable after provisioning, or reachable from some machines and not others depending on whose ARP cache won. ## Migrating a cluster built without a VIP A cluster provisioned before the VIP has its first control-plane node's address baked into the API server certificates, so this is not a config change you can sync. ### Rebuild (simplest) Reprovision from the [Quickstart](); the VIP is in place from `kubeadm init` onwards. Restore state per [Backups & Recovery](). ### In place One node at a time, with a second terminal holding a working kubeconfig, because several steps can leave you unable to open a new one. 1. Confirm the address is free. ```bash ping -c2 # must not answer arping -c2 # from a node on the segment ``` 2. Add the VIP under `apiServer.certSANs` in the stored `ClusterConfiguration`. ```bash kubectl -n kube-system edit configmap kubeadm-config ``` 3. Regenerate the API server certificate on **each** control-plane node. ```bash ssh core@ sudo mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.bak sudo mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.bak sudo kubeadm init phase certs apiserver sudo crictl ps | grep kube-apiserver # confirm it restarted ``` 4. Deploy the kube-vip static pod to each control-plane node, generated from the template's values or copied from a node provisioned with the new configuration, and verify the VIP answers. ```bash curl -k https://:6443/healthz ``` 5. Repoint the cluster: update `controlPlaneEndpoint` in the `kubeadm-config` ConfigMap, then on every node rewrite the server address in `/etc/kubernetes/*.conf` and `/var/lib/kubelet/kubeconfig` and restart the kubelet. 6. **Only now** change `k8sServiceHost` in `payload/platform/cilium/values.yaml` to the VIP and commit. The merge rolls every agent, so the VIP must answer before it lands; watch the rollout. ```bash kubectl -n kube-system rollout status ds/cilium ``` 7. Re-fetch your kubeconfig so it uses the VIP. ```bash make kubeconfig ``` > [!CAUTION] > > **Danger** > > Cilium replaces kube-proxy, so if `k8sServiceHost` names an address that is not answering, every Cilium pod loses the API server and the cluster's networking goes with it, including the networking you would fix it through. Do not commit step 6 until `curl -k https://:6443/healthz` succeeds. ## Verifying ```bash # which node currently holds the VIP kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}'; echo kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip -o wide # the endpoint your kubeconfig actually uses kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo ``` Test failover once, deliberately: reboot the leader per [Rebooting a node]() and confirm `kubectl` keeps working after a few seconds. # Encryption Key Rotation The API servers reload `/etc/kubernetes/enc/encryption-config.yaml` whenever it changes (`encryption-provider-config-automatic-reload` in `ansible/templates/kubeadm.yaml.j2`), so a rotation is a file edit on each control plane and one rewrite of every Secret; what the key protects and how it is generated is in [Encryption at rest](). The first key in the list encrypts, every key in the list decrypts, and a Secret's ciphertext names the key it was written with. Two things follow: the new key goes in *above* the old one, and the cluster has to end up with the new key under the name the Butane template renders (`key1` in `ansible/templates/butane_node_config.yaml.j2`), because a control plane provisioned later gets exactly that file and can read nothing written under another name. That is why the Secrets are rewritten twice. Reading or writing the file from a shell fires `TetragonSensitiveFileAccess`; that is the alert doing its job, not a problem to fix. ## Runbook 1. Generate the new key beside the old one, in the shape `config.yaml`'s `password` lookup produces: 32 letters and digits, stored in plain text, and base64-encoded where it goes into the config. ```bash LC_ALL=C tr -dc 'A-Za-z0-9' output/credentials/encryption_key.new base64 < output/credentials/encryption_key.new # the secret for key2 below ``` 2. On every control plane, add the new key as `key2` **above** `key1` in `/etc/kubernetes/enc/encryption-config.yaml`; the rest of the file stays as it is. ```yaml providers: - secretbox: keys: - name: key2 secret: - name: key1 secret: - identity: {} ``` 3. Wait for every API server to reload. Each reloads its own file, so ask each one by its `ansible_host` rather than through the VIP; a `status="failure"` count means the file did not parse and the old config is still in effect. ```bash kubectl get --raw /metrics --server https://:6443 | grep apiserver_encryption_config_controller_automatic_reloads_total ``` 4. Rewrite every Secret so it is encrypted with `key2`. A conflict on a Secret something updated mid-stream is harmless: run it again. ```bash kubectl get secrets -A -o json | kubectl replace -f - ``` 5. On every control plane, replace `key1`'s secret with the new key and move it back above `key2`; both entries now hold the new key. Wait for the reload as in step 3. ```yaml keys: - name: key1 secret: - name: key2 secret: ``` 6. Rewrite every Secret again, so every ciphertext names `key1`. ```bash kubectl get secrets -A -o json | kubectl replace -f - ``` 7. On every control plane, remove `key2`. The file now matches what Ignition writes for a new control plane, and nothing on any node holds the old key. ```yaml keys: - name: key1 secret: ``` 8. Make the new key the one `make config` renders, and verify that the Butane config of every control plane carries it as `key1`; `config.yaml` reads the file back instead of generating a new value, so nothing else changes. The Ignition JSON cannot be grepped, since Butane gzips file contents, so the check reads the Butane YAML it was transpiled from in `output/tmp/`. ```bash mv output/credentials/encryption_key.new output/credentials/encryption_key make config && grep -c "$(tr -d '\n' < output/credentials/encryption_key | base64)" output/tmp/butane/butane-*.yaml ``` One match per control plane, none per worker or installer config. 9. Confirm against etcd that a Secret's ciphertext names the new key, with the same `etcdctl` flags as [Taking one by hand](); the value starts with `k8s:enc:secretbox:v1:key1:`. ```bash kubectl -n kube-system exec etcd- -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get /registry/secrets/kube-system/ --prefix --limit 1 | head -c 120 ``` Store the new key with the rest of `output/credentials/` wherever the old one is kept, and keep the old one next to the etcd snapshots taken before the rotation: those restore only with the key they were written under. # Host Firewall Inventory 2026-10-30 `ss -lntu` and the established peers of each listening port on all ten nodes, read before the [host firewall]() selected any of them. "Cluster" is every node and pod; "LAN" is everything else, Cilium's `world`. ## Listening on a node address | Port | Listener | Nodes | Connections seen from | Admit from | | --- | --- | --- | --- | --- | | 22/TCP | sshd | all | Workstation | LAN, cluster | | 6443/TCP | kube-apiserver, kube-vip's address on the holder | control plane | Workstation, every node, pods | LAN, cluster | | 2379/TCP | etcd client | control plane | Loopback only | Cluster, node range | | 2380/TCP | etcd peer | control plane | The other members | Cluster, node range | | 2381/TCP | etcd metrics, plain HTTP | control plane | Prometheus | Cluster | | 10250/TCP | kubelet | all | API server, Prometheus, metrics-server | Cluster | | 10257/TCP, 10259/TCP | controller-manager, scheduler | control plane | Prometheus | Cluster | | 4240/TCP | Cilium health | all | Agents and health endpoints | Cluster | | 4244/TCP | Hubble peer | all | Hubble Relay | Cluster | | 9100/TCP | node-exporter | all | Prometheus | Cluster | | 9962/TCP, 9965/TCP | Cilium agent and Hubble metrics | all | Prometheus | Cluster | | 9963/TCP | cilium-operator metrics | The two operator nodes, moves with the pod | Prometheus | Cluster | | 9964/TCP | Envoy metrics | all | Nothing; the Prometheus egress policy does not name it | Cluster | | 2112/TCP | Tetragon metrics | all | Prometheus | Cluster | | 6789/TCP | Tetragon | all | Nothing | Cluster | | 8472/UDP | VXLAN | all | Other nodes | Cluster | | 51871/UDP | WireGuard | all | Other nodes | Cluster | | 68/UDP | DHCP client | all | Router | LAN | | ICMP echo | Kernel | all | Not measured | LAN, cluster | ## Not filtered, or not listening | Traffic | Why no rule | | --- | --- | | Loopback: 2379, 2381, 10248, 9878, 9879, 9890, 8118, 30003, DNS on 127.0.0.53 and .54, Envoy's and containerd's dynamic ports | Loopback never crosses a device the firewall is attached to | | NodePorts of the two Gateway Services | Translated to the backend before the firewall | | The two Gateway addresses | Not node addresses; the firewall checks only packets addressed to one | | BGP | The agent dials the router and nothing listens on 179, so only replies arrive | | Ceph mon and OSD ports | Rook runs them on the pod network; the only host-networked Ceph pod, the RBD node plugin, listens on nothing | | DHCP server, TFTP, PXE | The boot server is off-cluster; a node's PXE boot happens before any agent runs | | kube-vip | ARP mode: gratuitous ARP passes the firewall, and leader election goes through the API server | ## Findings | Finding | Consequence | | --- | --- | | Services bound to `*` answer on each node's global IPv6 address: `curl` on port 9100 from the workstation returned `200` | The firewall cannot filter IPv6 with `enable-ipv6: false`; it [drops it all](). No IPv6 connection other than loopback was established on any node; each node's second resolver is IPv6, used only after the first fails | | Prometheus's scrapes arrive from its node's address, not the pod's | Masquerading, see [Host firewall](). Prometheus-only is held by the pod egress policies: besides the API server rules, only Prometheus, Hubble Relay and metrics-server name a node as a destination, each with its ports | | Tetragon holds 2112 on the control planes, the port kube-vip also wants | kube-vip logs `bind: address already in use`, and its liveness probe is answered by Tetragon | | 6789 is Tetragon, not a Ceph monitor | Nothing on a node needs a Ceph rule | # Network Policy Audit 2026-09-27 One week of `AUDIT` verdicts from every namespace, read at the end of the [rollout](), kept so the flip to enforcement starts from the disposition rather than the raw count. 25k verdicts, 21k of them one missing pair of rules. ## Rules added | Flow | Disposition | | --- | --- | | Prometheus and the CloudNativePG operator to Authentik's database on 9187 and 8000 | **Rule.** The database moved in \#906 and the policy's spec never followed its description: \#927 | | Every `http: [{}]` with a machine caller | **Tightened** to the methods and paths the week recorded: `GET /metrics` for every scrape, the four calls External Secrets makes at OpenBao, Loki's `POST /api/v2/alerts`, Grafana's `GET /loki/api/v1/…`: \#928, apps \#59. UIs behind the Authentik outpost stay open | ## Config switches | Flow | Disposition | | --- | --- | | Authentik server to `goauthentik.io`, twice per pod start | **Switched off.** `disable_startup_analytics`: \#929 | | Authentik server to `www.gravatar.com`, 117 times | **Not a switch the repository holds.** `AUTHENTIK_AVATARS` seeds the tenant once; the live value is a system setting and the database still says `gravatar,initials`. Set `initials` under System → Settings; the upstream ask is [goauthentik/authentik\#26470](), the PR [\#26471]() | | Nextcloud to `updates.nextcloud.com` and `pushfeed.nextcloud.com` | **Switched off** in the startup hook: apps \#60 | | Trivy scan jobs to `check.trivy.dev`, Authentik worker to `version.goauthentik.io`, Alloy and Loki to `stats.grafana.org`, Grafana to `grafana.com` and `secure.gravatar.com` | **Already off**, by \#861, \#883 and the values that predate the week; every record is from before the switch landed | ## Nothing to do | Flow | Why | | --- | --- | | A name a `toFQDNs` rule allows, one `SYN` audited now and then: Alertmanager to the mail relay 14 of 254 mails, the repo-server to `release-assets.githubusercontent.com`, Home Assistant to `alerts.home-assistant.io`, scan jobs to the registries | The destination identity in the record is `world`, so the IP was not yet, or no longer, in the name's set when the `SYN` left. Eight fresh lookups from the Home Assistant pod were all allowed. Under enforcement the first `SYN` drops and the retransmit a second later decides it, so the `DROPPED` records after the flip say which: one drop per connection is the DNS proxy's 100ms hand-off, a run of drops on one source port is an application connecting to an address it cached past the TTL | | Replies on high ports, pod addresses classified `world`, `openbao-N.openbao-internal` during a restart | Answer halves and identities of pods that had just gone; the export marks the first `is_reply` | | `ceph-object-controller-detect-version` to its own node's API server, 3 times on thor | The host identity there carries `reserved:kube-apiserver`, and the same Job reached the other control-plane nodes hundreds of times; not at an agent restart. Watch for it as `DROPPED` | | Home Assistant to the node's `cilium_host` address on UDP 53, 3 times; SSDP to `255.255.255.255` before apps \#44; `argocd-redis-secret-init` before \#871 | Predate their fix, or too rare to name | ## Consolidation Not done, recorded: `argocd-server` and `argocd-server-oidc` select the same pods and could be one policy; the intra-namespace pair and the node rule are in 18 of 19 namespaces and are the next cluster-wide candidates after DNS, at the cost of the per-namespace file no longer being the whole record. The Prometheus rule is worded identically in 17 files, which is the point. # Network Policy Audit 2026-10-16 Two weeks of `AUDIT` verdicts, 195k records, read before `policyAuditMode` went off. The method is the [rollout](); the first reading is [Network Policy Audit 2026-09-27](). No namespace with pods is without a policy, and every pod endpoint on the nodes checked is selected in both directions. ## Fixed | Flow | Disposition | | --- | --- | | Velero and its data movers to `s3.infra.k8s.wlkr.ch`, 3.7k requests a day since 2026-10-04 | **Rule.** Not an audit verdict: the Gateway's Envoy checked Velero's egress against the RGW pod and answered `403` itself, which audit mode never sees. The backup location was Unavailable and every daily backup `FailedValidation`. Egress to the RGW pod on 8080 in `backup.yaml` | | Pods in their first seconds, under the `init` identity: Trivy scan jobs' lookups and their calls to the Trivy server, the kubelet's probes, lookups landing on a CoreDNS pod that had just started; 27k records | **Cluster-wide policy** `init`: the node, DNS out and DNS in. A scan job's call to the Trivy server in the same seconds retransmits until the identity arrives | | Every header of every proxied request in the export, `X-Vault-Token`, bearer tokens and cookies included | **Redacted** with `hubble.redact`; see [Cilium](). Records written before the change keep them until Loki's retention drops them | ## Nothing to do | Flow | Why | | --- | --- | | Rook mons, mgrs and crash collectors to `10.108.70.163`, `10.108.19.154` and `10.102.93.197` on 3300, 50k records from 2026-10-13 to the restarts of 2026-10-16 | Service addresses of mons that no longer exist; the daemons kept them until they restarted. A connection to an address with no backend fails with or without a policy | | OSDs and RGW to pod addresses on 6800–6808 classified `world`, after an OSD restart | Addresses of pods that had just gone | | The CloudNativePG operator to every instance on 8000, a few dozen at a time, and other in-cluster flows at agent restarts and node joins (2026-10-10, 2026-10-16) | The rule exists; the peer's identity had not reached the agent yet | | A name a `toFQDNs` rule allows, one `SYN` now and then: Authentik to `www.gravatar.com`, the repo-server to the chart repositories, Dependency-Track to its package registries, scan jobs to the registry CDNs, Alertmanager to the mail relay | The DNS proxy's hand-off, as in the first reading. Gravatar stays a database setting | | Prometheus and metrics-server to the five new nodes as `world`, 2026-10-10 | The nodes joining | | Rook's mgr and detect-version Job to a control-plane node's API server as `host`, a handful a day | The node identity briefly without `reserved:kube-apiserver`; watch for it as `DROPPED` | ## Decisions left open | Flow | Question | | --- | --- | | Agent sessions to Prometheus, Hubble Relay, OpenBao and Argo CD in the cluster, and to PyPI, npm, `dl.k8s.io` and `get.helm.sh` | Each session's reach is its chart's toggles, so these now drop. Granting one is a change to the session, not to a namespace policy | | Agent sessions to `http-intake.logs.us5.datadoghq.com` | The `claude-agent` chart allows Claude Code's error reporting by name; telemetry here is switched off at the source, not allowed | # Node Lifecycle Everything you do to one machine. The four procedures share a shape: drain, act, wait for Ceph, move on. > [!CAUTION] > > **Two of these destroy data and two do not** > > [Rebooting](<#rebooting-a-node>) and [adding](<#adding-a-node>) are safe and routine. [Replacing](<#replacing-a-failed-node>) destroys one node's OSD. [Rebuilding](<#rebuilding-or-repartitioning-a-node>) destroys the disk it runs against, and doing it to every node in turn destroys every replica of everything. ## Rebooting a node [Kured]() reboots nodes on its own once an update is staged, one at a time, and refuses while Ceph or etcd is unhealthy. The procedure below is for rebooting ahead of its next check (`period` in `payload/platform/kured/application.yaml`), or for a reason nothing set a sentinel for. Flatcar is installed to disk, so the node boots without the boot server and keeps `/etc/kubernetes`, `/var/lib/etcd` and `/var/lib/rook`. A change made with `make config` is *not* picked up by a reboot; that takes a [rebuild](<#rebuilding-or-repartitioning-a-node>). 1. Drain it. ```bash kubectl drain --ignore-daemonsets --delete-emptydir-data ``` A drain that loops on `Cannot evict pod as it would violate the pod's disruption budget` for a `-db-` pod is a CloudNativePG cluster with its budget still on — see [the contract](). 2. Reboot it and wait for `Ready`. ```bash ssh core@ sudo systemctl reboot kubectl get node -w ``` 3. Uncordon it. ```bash kubectl uncordon ``` The pods that moved stay where they are; the [descheduler]() spreads them back out on its next pass. 4. **Wait for Ceph** before touching the next node: draining a second node while the first is still backfilling can take a placement group below its minimum replica count. ```bash kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status # HEALTH_OK ``` 5. **Unseal OpenBao** if the node hosted a replica (below). ## After any node reboot: unseal OpenBao OpenBao comes back sealed and looks healthy while sealed, and nothing unseals it for you — see [OpenBao]() for what that costs. ```bash make bao-unseal # idempotent: unseals whatever is sealed for pod in openbao-0 openbao-1 openbao-2; do kubectl -n openbao exec "$pod" -- bao status | grep Sealed # false done ``` `make bao-unseal` reads the key shares from `output/credentials/openbao-init.json`. If that file has been moved into a password manager, each pod wants the threshold of shares by hand, per [OpenBao](). ## Adding a node 1. Compare `kubernetes_version` in `ansible/inventory.yaml` with `kubectl version`: a node installs the inventory's version, and `kubeadm join` refuses a kubelet newer than the API server, so a merged minor has to be [walked onto the cluster]() first. Then add the host to the inventory and regenerate and serve the boot files. ```bash make config && make serve ``` In the same commit, add the node name to `providerRegex` in `payload/platform/kubelet-csr-approver/application.yaml`, and let that PR merge and sync before the node boots: a node the regex does not name has its kubelet certificate denied and [silently drops out of `kubectl top`](). Against a running cluster, `make config` also re-creates the bootstrap token baked into the Ignition configs for 24 hours and uploads the control-plane certificates for two, on [whichever control plane answers](), so the node joins on its own. 2. Arm the node: the generated menu defaults to the local disk, so an unarmed node never installs, per [Switching back to local boot](). ```bash make reinstall LIMIT= ``` 3. Network-boot it within those windows: two hours for a **control-plane** node, 24 for a worker. `bootstrap-k8s.service` runs the join once and only while `/etc/kubernetes/kubelet.conf` is absent, so it does not interfere with a node that has joined. 4. If the window has passed, run `make config` again, then retry the join on the node. ```bash ssh sudo systemctl restart bootstrap-k8s ``` ## Replacing a failed node 1. Remove it from the cluster. ```bash kubectl drain --ignore-daemonsets --delete-emptydir-data --force kubectl delete node ``` 2. Let Ceph re-replicate. With `useAllNodes: true` the OSD on that disk is gone for good; wait for `ceph status` to return to `HEALTH_OK`. 3. If it was a control-plane node, remove its etcd member; a dead member still counts toward quorum. ```bash kubectl -n kube-system exec -it etcd- -- etcdctl \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ member list # then: member remove ``` 4. Reprovision the replacement per [Adding a node](<#adding-a-node>): the inventory and regex change, then arm, then boot. > [!CAUTION] > > **Danger** > > On a cluster provisioned before the [Control Plane VIP](), `odin` is not an interchangeable control-plane node: its address is baked in as the API endpoint and as Cilium's `k8sServiceHost`, so losing it breaks node joins and Cilium's API connection everywhere. Check which endpoint your kubeconfig uses first. ## Rebuilding or repartitioning a node The disk layout in `ansible/templates/butane_node_config.yaml.j2` is applied by Ignition once, on the first boot after install, and the Ignition config is embedded in the OEM partition at install time, so no change under `ansible/` reaches a running node. `rook-osd` is the last partition and deliberately raw: move its start by a sector, which any insert or resize above it does, and every OSD on that node is gone. Only the last partition can grow without a reinstall, and `rook-osd` cannot shrink in place because Ceph has already written across the space. > [!CAUTION] > > **The backups are inside the thing being wiped** > > Velero and the etcd snapshot CronJob both write to the Ceph object store this destroys — see [Backups & Recovery](). Copy anything you intend to restore from **off-cluster** first. 1. Copy what matters off the cluster: the `velero` and `etcd-backup` buckets per [Copying a bucket off the cluster](), and anything in a PVC not reproducible from Git. 2. Edit whatever needs editing under `ansible/`. 3. Regenerate both Ignition configs and serve them. ```bash make config && make serve ``` 4. Drain the node, as in [Rebooting a node](<#rebooting-a-node>). For a **control-plane** node, remove it from the cluster too: the `kubectl delete node` and the etcd `member remove` are steps 1 and 3 of [Replacing a failed node](<#replacing-a-failed-node>). The wiped node comes back with an empty etcd under its old peer URL, and kubeadm treats a known peer as already joined; two of three members keep quorum meanwhile. 5. Arm the nodes you are rebuilding. Arming is the only way in (the menu shows no prompt), a later `make config` keeps a node armed, and the boot server disarms it once it has the image, so the reboot at the end of the install does not start a second one — see [Switching back to local boot](). ```bash make reinstall LIMIT=odin # `make reinstall` alone asks, then arms every host ``` Every control plane joins the running cluster; only [`bootstrap_cluster`]() renders `kubeadm init`, and arming refuses while it is set and the VIP answers. 6. Network-boot the node. The installer wipes the disk, runs `flatcar-install`, and reboots into the installed system, which then runs `kubeadm`. The node comes back with a new SSH host key; `make reinstall` already forgot the old one, or `StrictHostKeyChecking=accept-new` would refuse the first `make kubeconfig`. 7. Leave the boot server up until the node is `Ready` (the sysext images are fetched from it on that first boot), then stop it. 8. Wait for Ceph before the next node, as in [Rebooting a node](<#rebooting-a-node>): the OSD is new and backfills from the other two copies, and [`make storage-check`]() proves it took the disk. 9. [Unseal OpenBao](<#after-any-node-reboot-unseal-openbao>) if a replica moved. One node at a time is safe when rebuilding: etcd keeps quorum once the old member is removed and Ceph backfills, as in [Replacing a failed node](<#replacing-a-failed-node>). Repartitioning destroys every OSD as it goes, so a rolling repartition across every node destroys all replicas of everything; step 1 is not optional for that case. # Triage 2026-09-20 The result of one pass over the Trivy Operator reports, kept so the next one starts from the disposition rather than the raw count. The method is in [Vulnerability Triage](). ## Images Every image with a critical finding or a fixable high. | Image | Finding | Disposition | | --- | --- | --- | | CSI sidecars (`csi-provisioner`, `csi-resizer`, `csi-snapshotter`) | gRPC-Go authorization bypass, CVE-2026-33186 | **Fixed here.** Pinned to the versions ceph-csi-operator v1.0.5 defaults to; Rook still ships v1.0.4 — see [Rook-Ceph → CSI driver](). The ask to bump Rook is drafted for a person to post | | `argoproj/argocd` v3.5.3 | Go stdlib and `x/net` in bundled `kustomize` (Go 1.24.0) and `git-lfs` 3.7.1; `grpc`, `oras-go` in argocd | **Waiting on release.** `master` has git-lfs 3.8.0, grpc 1.83.2, oras-go 2.6.2; only a new kustomize build fixes kustomize | | `dexidp/dex` v2.45.1 | OpenSSL in Alpine 3.23, `grpc`, `goxmldsig`, stdlib; a second set in bundled `gomplate` | **Waiting on release.** Fixed on `master`, nothing released since March 2026: [dexidp/dex\#4948](), scan data added 2026-09-18 | | `rook/ceph` v1.20.7 | Go stdlib in bundled `s5cmd` 2.3.0 (Go 1.22.10) | **Not reachable, waiting on release.** Nothing in Rook invokes `s5cmd`; [peak/s5cmd\#873](), [\#820]() ask for a release | | `home-assistant/home-assistant` | OpenSSH in Alpine 3.24.1, Go stdlib in bundled `tempio` (Go 1.23.3) and `go2rtc`, Python `anyio` | **Waiting on release.** The base image already carries tempio 2026.07.0; weekly releases. Nothing filed, by their policy | | `library/postgres` 17.11 (Authentik) | Go stdlib in `gosu` 1.19; `libxml2` in Debian 13, no fix | **Not reachable.** The stdlib CVE is TLS session resumption and gosu opens no connections; libxml2 waits on Debian | | `cloudnative-pg/postgresql` 18.6 | 31 highs and one critical, none with a fix (Debian 13) | **Nothing to do.** Base is current; every finding waits on Debian | | `coredns/coredns` v1.14.6 | Two highs fixed in 1.14.7, both on DoH, DoQ and gRPC listeners | **Not reachable.** kubeadm's Corefile serves plain UDP and TCP; kubeadm 1.37 pins 1.14.6 | | `grafana/grafana` | `grpc`, `otel`, `x/net` in thirteen bundled datasource plugin binaries | **Waiting on release.** Rebuilt by Grafana's own release; 13.2.2 is current | | `ceph/ceph` v20.2.4 | `setuptools` 69.2 under Python 3.9 | **Not reachable.** The CVEs are in `easy_install` and `package_index`, which no daemon calls | | `openbao/openbao` 2.6.2 | `github.com/openbao/openbao` "fixed in 2.5.4" | **Scanner artifact.** A Go pseudo-version sorts below every real tag | | `library/nextcloud` 34.0.4 | 2887 findings, 2886 in Debian packages with no fix; one fixable LOW in bundled `webauthn-lib` | **Nothing to do.** Base is current Debian 13; `webauthn-lib` is pinned `^4.9.1` upstream too. The report exists only because of the per-namespace policy (homelab-apps\#28) | | `goauthentik/server` 2026.8.3 | `anyio` 4.14.1 (critical), `msgpack`, `setuptools`; `libxml2` no fix | **Waiting on release.** `uv.lock` on `main` already has the fixed versions; monthly releases | | `xperimental/nextcloud-exporter` 0.9.1 | Go stdlib, 28 advisories, 7 HIGH (Go 1.26.1) | **Filed:** [xperimental/nextcloud-exporter\#143](). No bot bumps the builder tag | | `prometheus-config-reloader` v0.91.0 (Alloy's sidecar) | Go stdlib, 4 HIGH | **Waiting on chart release.** `main` carries v0.94.0 | | Everything else | `grpc` one or two patches behind, Go stdlib one patch behind | **Routine churn.** Not a finding | Scan coverage: Nextcloud and Authentik scans were dying on a five-minute Job deadline, and their reports were then too large for etcd; fixed in homelab-apps \#26 and \#28. The two exposed-secret findings are artifacts: `ssl-cert-snakeoil.key` in the Postgres pods is Debian's placeholder, and the "Azure Storage Account Key" in Nextcloud is the public Azurite development key (ignored by homelab-apps\#31). ## Infra: the nodes The node collector runs CIS's file checks on every node. | Check | Path | Was | Disposition | | --- | --- | --- | --- | | KCV-0077 kubelet config | `/var/lib/kubelet/config.yaml` | 644 | **Fixed**, 600 | | KCV-0075 CA file | `/etc/kubernetes/pki/ca.crt` | 644 | **Fixed**, 600 | | KCV-0056 CNI files | `/opt/cni/bin` | 755 | **Fixed**, 700 | | KCV-0069 kubelet unit | `/lib/systemd/system/kubelet.service` | 644 | **Accepted.** Ships in the signed, read-only sysext and holds no secret | | KCV-0059 etcd data dir | `/var/lib/etcd/default.etcd` | absent | **Artifact.** kubeadm uses `/var/lib/etcd`, already `700 etcd:etcd` | | KCV-0001 and the other API server flags | static pod args | | **Tracked in \#660.** Anonymous auth cannot simply go: the kubelet's probes rely on it (\#678) | kubeadm rewrites the first three at 644 on every `init`, `join` and `upgrade`, so `ansible/templates/butane_node_config.yaml.j2` ships a `systemd-tmpfiles` rule that the kubelet drop-in re-applies before every start. It is provisioning-time: a node picks it up on its next rebuild. ## Config audit: the pods The permanent exceptions are in [What is load-bearing](). Changed on this pass: | Where | Disposition | | --- | --- | | Argo CD, External Secrets, Trivy Operator, metrics-server, kubelet-csr-approver, snapshot-controller, Alloy | **`KSV-0118` only, fixed** (\#791; homelab-apps\#27 for Trivy Operator). Containers already ran non-root; the pod-level context was empty. Kured's chart has no pod-level value; CoreDNS is kubeadm's | | Authentik | **Fixed** (\#792): non-root, no capabilities, no escalation, runtime seccomp. Read-only root not attempted: the image writes under `/media` and `/templates` | | Nextcloud, Home Assistant | **Seccomp only** (homelab-apps\#30) | | Velero | **Node agent and plugin init container confined** (\#794); the `velero` container is unchanged pending a real backup — see [Velero]() | | trivy-server, nextcloud-exporter | **Fixed** (homelab-apps\#32, \#31) | | `etcd-backup` CronJob | **Own manifest, hardened** | Of the medium count, `KSV-0125` and the root, escalation and seccomp findings in `rook-ceph` and `kube-system` are the load-bearing set; outside those two namespaces the remaining rows are the ones above. Nine `ConfigAuditReport`s belonged to superseded ReplicaSets; the Deployments this repository controls now keep one or two (homelab-apps\#33), and Nextcloud's chart has no value for it. ## RBAC Every critical and high is a chart's operator `ClusterRole` or Kubernetes' own `admin`, `edit` and `cluster-admin`. Nothing filed, nothing changed. # Triage 2026-09-29 The first pass over the [Dependency-Track]() portfolio, read next to the Trivy Operator reports. The method is in [Vulnerability Triage](); the image dispositions that did not change are in [Triage 2026-09-20](). ## The data | Finding | Disposition | | --- | --- | | Projects from ReplicaSets scaled to zero, ten of 76, with a quarter of the criticals | **Fixed** (homelab-apps\#77, \#983). The upload job skips them | | `isLatest` on a superseded version | **Fixed** (homelab-apps\#77). It goes to the highest version that runs | | No project for `busybox` and `hubble-export-stdout` | **By design.** Neither image has a package Trivy can name, and the operator writes no `SbomReport` for an empty SBOM | | No project for Home Assistant | **Accepted.** Its SBOM names over three thousand components and is larger than one API write may be, so the operator cannot store it; see [Pitfalls](). The `VulnerabilityReport` covers the image | | Most findings carry no severity | **By design.** They are Go vulnerability database entries, which have none; the same flaw is usually there a second time as a GitHub advisory that does | ## Images Every critical on an image that runs. Dependency-Track rates by the advisory's own score, Trivy by the distribution's, so the two disagree on severity more often than on presence. | Image | Finding | Disposition | | --- | --- | --- | | Argo CD (bundled `git-lfs`), dex, snapshot-controller, descheduler, cephcsi, csi-attacher | Seven `golang.org/x/crypto` advisories in its SSH server, client and agent | **Not reachable, waiting on release.** None of them speaks SSH. Trivy has all seven at severity `UNKNOWN`, so they never reached its dashboard | | dex, snapshot-controller | gRPC-Go authorization bypass, CVE-2026-33186 | **Not reachable, waiting on release.** dex as on 2026-09-20; snapshot-controller is a client, and its image trails the tag by one release, see [Velero]() | | `library/nextcloud` | Eight in `curl`, five in `rsync`, one in `libxml2` | **Nothing to do.** No fix in Debian 13. The `rsync` ones need a daemon or a remote shell, and the image copies files locally. Trivy rates the `curl` ones below HIGH, where the namespace's ignore policy drops them | | `library/nextcloud` | `MAL-2022-3033`, `MAL-2022-3056`, "malicious package" | **Scanner artifact.** Nextcloud's own `files_pdfviewer` and `firstrunwizard` apps share a name with two npm packages. **Fixed** (homelab-apps\#78): the upload job suppresses them for every new tag | | `goauthentik/server`, `cloudnative-pg/postgresql` | `libxml2`, `anyio` | **Unchanged** since 2026-09-20 | | `dependencytrack/apiserver` | FreeMarker template path traversal | **Waiting on release.** Needs a caller who chooses the locale; the templates here are the notification ones | | `janwelker/knead-time-push` | OpenSSL, and in Trivy also `perl`, `sqlite` and `zlib`, none fixed in Debian 12 | **Fixed** (knead-time\#371, homelab-apps\#79). The image moves to a distroless base, which has none of the four packages but OpenSSL's libraries | What Dependency-Track does not rate critical and Trivy does: the Go standard library in bundled `kustomize`, `git-lfs`, `gomplate` and `s5cmd`, dex's Alpine OpenSSL, and Nextcloud's `libopenexr`. Neither source is the whole list. ## The operator Found on the way: the operator was being OOM-killed several times a day, because its cache holds every report and the `SbomReport`s are large. A kill while a report is written loses it without an error. Fixed in \#985. # Troubleshooting The thing that looks broken is usually downstream of something duller. Start with the table. ## Where to look first | Symptom | First check | | --- | --- | | Secrets missing, certificates not renewing | OpenBao sealed — `kubectl -n openbao exec openbao-0 -- bao status`; if so, `make bao-unseal` | | An Application is Degraded, OutOfSync or stuck Progressing | `argocd app get ` for the failing resource; a failed sync [retries itself for about forty minutes](), after that `argocd app sync ` — see [ArgoCD]() | | A node is down | [Rebooting a node](); if it does not come back, [Replacing a failed node]() | | Pods pending on a fresh node | Node still `NotReady`; check Cilium is running on it | | `ExternalSecret` not syncing | [External Secrets troubleshooting]() | | PVCs stuck pending | `ceph status`, then the [Rook dashboard]() | | A hostname stopped resolving to the cluster | Gateway lost its LoadBalancer IP; check the Cilium L2 pool | | Node not rejoining after reboot | `journalctl -u kubelet` on the node | | A node will not install, or hangs at a blinking cursor | [Troubleshooting PXE boot]() | | A node installs but comes up with no OSD | [Rook-Ceph → No OSDs after reprovisioning]() | Alerts are mailed by Alertmanager — see [Monitoring → Alerting]() — but the delivery path itself is not monitored, so an empty inbox does not prove anything. ## Rolling back a bad sync The only fix is a revert commit. Every Application runs with automated sync, so ArgoCD refuses `argocd app rollback`, and a manual sync to an older revision is undone within one polling interval. Do not patch a generated Application's `spec` to switch automated sync off either; the next ApplicationSet reconcile copies the file back over it — see [Sync policy](). The exception is `argocd` itself, which is not generated: `argocd app set argocd --sync-policy none` holds a rollback until `syncPolicy.automated` is restored. > [!WARNING] > > Do not fix a broken workload with `kubectl edit`. `selfHeal` on every Application ([Sync policy]()) puts the manifest back on the next reconcile, so the fix lasts minutes and leaves no trace of what it was. Change the manifest instead. # Updates & Upgrades Three mechanisms update a node, and all three land their changes on disk and wait for a reboot. A node can be four patch releases behind while reporting that everything is up to date. ## OS updates Flatcar downloads OS updates in the background into the passive half of its A/B partition pair. `locksmithd`, which would normally coordinate the reboot, is masked in `ansible/templates/butane_node_config.yaml.j2`, because a reboot must drain the node first. In its place `flatcar-reboot-sentinel.timer` polls `update_engine_client -status` and touches `/run/reboot-required` once an update is staged. [Kured]() consumes that marker: it drains, reboots and uncordons, one node at a time and only while Ceph and etcd are healthy. To move sooner, see [Rebooting a node](). Three alerts in `payload/platform/monitoring/node-update-rules.yaml` watch that this actually happens, because Trivy scans containers and not the host: `NodeRebootPending` when a node has carried the marker for longer than its `for`, `NodeOsVersionDrift` and `NodeKubeletVersionDrift` when the nodes have run different Flatcar or kubelet versions for longer than theirs. ```bash ssh core@ 'cat /etc/os-release; systemctl status update-engine --no-pager' ``` ## Kubernetes Kubernetes is a [systemd sysext](). `systemd-sysupdate.timer` runs an update for it on every fire and touches the same `/run/reboot-required` when a new image is fetched. containerd is part of the OS image and updates with it. ### Nodes are pinned to a minor series The sysupdate config points at `https://extensions.flatcar.org/extensions/`, not at the boot server, and its `MatchPattern` names the major.minor from `kubernetes_version` in `ansible/inventory.yaml` with only the patch left as the `@v` wildcard — see `ansible/templates/sysupdate_kubernetes.conf.j2`. Patch releases inside the series are picked up automatically; the next minor is not, because an unattended minor jump leaves kubelets ahead of a control plane that refuses to talk to them. sysext-bakery publishes a floating `kubernetes.conf` (`kubernetes-@v-%a.raw`) as well, which would stage a minor kubeadm cannot skip to, on whichever node checks first. `ansible/templates/sysupdate_kubernetes.conf.j2` renders the pinned file from the inventory and the Ignition config inlines it, so the source of every later sysext is reviewed in Git rather than fetched; `make config` also writes it to `output/http/kubernetes.conf` for running nodes. ```bash ssh core@ 'ls -l /etc/extensions/ /opt/extensions/kubernetes/' ssh core@ cat /etc/sysupdate.kubernetes.d/kubernetes.conf # confirm the MatchPattern is pinned ``` ## Upgrading a minor version deliberately **kubeadm allows one minor at a time.** v1.34 to v1.37 is three passes through this procedure, each with its release notes read. A newly provisioned node skips all of it: it installs `kubernetes_version` directly and gets the pinned sysupdate config from the boot server. The order is upstream's: the first control plane, the other control planes, then the workers, because a kubelet may run older than the API server but never newer. The sysext swaps kubeadm and kubelet together at the reboot, so the sysupdate config goes to one group at a time: Kured (`concurrency` in `payload/platform/kured/application.yaml`) reboots one node at a time but in no particular order, and a config pushed to every node at once hands the workers a new kubelet before the API server has moved. 1. Update `kubernetes_version` in `ansible/inventory.yaml`. 2. Fetch the new sysext and regenerate the Ignition and sysupdate configs. ```bash make download && make config ``` 3. Stage the new sysext on the first control plane only. The sysupdate config was written once at install time and nothing rewrites it. The boot server is not needed: the node reboots from its own disk — see [Every boot after the first](). ```bash scp output/http/kubernetes.conf core@:/tmp/kubernetes.conf ssh core@ sudo mv /tmp/kubernetes.conf /etc/sysupdate.kubernetes.d/kubernetes.conf ssh core@ sudo systemctl start systemd-sysupdate ``` 4. Reboot it to pick up the sysext — see [Rebooting a node](). Kured would do it once sysupdate sets the sentinel, but by hand the next step follows at once. 5. On that node, run the kubeadm upgrade; the sysext swapped the binaries and nothing else. Always pass `--skip-phases addon/kube-proxy`, because a cluster built before `proxy.disabled` was set in `kubeadm.yaml.j2` still has `proxy: {}` in its stored `kubeadm-config`, and without the flag the upgrade redeploys `kube-proxy` beside Cilium; the flag is harmless on a cluster built since. Always pass `--patches /etc/kubernetes/patches` too: the static pods' memory requests and CoreDNS's spread constraint come from that directory and an upgrade rewrites the manifests without them otherwise. ```bash sudo kubeadm upgrade apply --skip-phases addon/kube-proxy --patches /etc/kubernetes/patches ``` 6. The other control planes, one at a time: step 3, then step 4, then `kubeadm upgrade node` with the same `--patches` flag. The reboot comes first because the sysext swaps the binaries only then, and `upgrade node` has to run the new kubeadm. It takes the version from the `kubeadm-config` that `upgrade apply` stored and rewrites the static pod manifests on this node. ```bash sudo kubeadm upgrade node --patches /etc/kubernetes/patches ``` 7. The workers, one at a time: step 3, then step 4, then `kubeadm upgrade node`, which only fetches the new kubelet configuration. ```bash sudo kubeadm upgrade node ``` > [!NOTE] > > Renovate keeps `kubernetes_version`, `flatcar_version` and `syslinux_version` current in `ansible/inventory.yaml`; which bumps automerge is in [Automerge policy](). ## Platform components Everything in `payload/` is upgraded by ArgoCD when Renovate bumps a `targetRevision` and the PR merges. Cilium, the Gateway API CRDs and ArgoCD are installed by `make install-cilium` and `make install-argo` before ArgoCD exists, but from the same `targetRevision`, so a rebuild lands where the cluster already is — see [Version pins](). ## Control-plane certificates kubeadm issues the control-plane certificates for one year and renews them only during `kubeadm upgrade`. Patch updates arrive as a sysext and never run kubeadm, and a minor is [a manual procedure](<#upgrading-a-minor-version-deliberately>), so a year without one takes the API servers, the etcd peers and the front proxy down together. The kubelet certificates are the exception: the serving certificate is rotated through [kubelet-csr-approver]() and the client certificate by the kubelet itself. `payload/platform/monitoring/control-plane-certificate-rules.yaml` warns thirty days ahead on the client certificates the API server sees, which kubeadm issues together with the serving ones; the chart's own `KubeClientCertificateExpiration` takes over at seven days and one day. The renewal is done by hand, once a year or when the alert fires, on every control plane in turn: 1. Read the remaining lifetime; every line but the CA should have well over thirty days. ```bash ssh sudo kubeadm certs check-expiration ``` 2. Renew everything kubeadm issued on that node. The CA is not touched. ```bash ssh sudo kubeadm certs renew all ``` 3. Restart the static pods so they load the new certificates: move the manifests out, wait until `crictl` lists none of the four, move them back. One node at a time, since the API server and etcd on it are down in between. ```bash ssh 'sudo mkdir -p /root/manifests-stopped && sudo mv /etc/kubernetes/manifests/*.yaml /root/manifests-stopped/ && until [ -z "$(sudo crictl ps -q --name "kube-apiserver|kube-controller-manager|kube-scheduler|etcd")" ]; do sleep 2; done && sudo mv /root/manifests-stopped/*.yaml /etc/kubernetes/manifests/' ``` 4. After the last node, refresh the admin kubeconfig: `admin.conf` was renewed too, and the copy in `output/kubeconfig` carries the old client certificate. ```bash make kubeconfig ``` # Vulnerability Triage Trivy Operator writes one `VulnerabilityReport` per container, a few thousand findings on a cluster this size. Almost none warrant action, and the work is in proving that quickly. The operator's configuration and failure modes are in [Trivy Operator](). This page is the method; the dispositions are in [Triage 2026-09-20]() and, for the Dependency-Track portfolio, [Triage 2026-09-29](). ## Pulling a report ```bash kubectl get vulnerabilityreports -A -o json > vr.json jq -r '.items[] | .metadata.namespace as $ns | .report.artifact.repository as $img | .report.vulnerabilities[]? | select(.severity=="CRITICAL") | [$ns, $img, .vulnerabilityID, .resource, .installedVersion, (.fixedVersion // "NO-FIX"), (.packagePath // .target)] | @tsv' vr.json | sort -u ``` ## Reading it - **A missing report is not a clean image.** A failed scan writes nothing; the `TrivyContainerNotScanned` alert reconciles running images against scanned ones continuously. - **An empty `fixedVersion` means no fix is known, not no problem.** Unfixed findings are kept (`ignoreUnfixed` in `payload/platform/trivy-operator/application.yaml`) because a cluster of them in one base image says the base image is the wrong one. - **The worst findings are usually not the project's code.** The `target` names the binary, and which binary decides whether anything can be done and by whom. - **Severity is an opinion, and the two sources hold different ones.** [Dependency-Track]() rates by the advisory's score and Trivy by the distribution's; a finding Trivy has at `UNKNOWN` can be critical there. Use it for which images share a component and for what changed between two tags. - **Prometheus is not a trend.** It keeps the `retention` in `payload/platform/monitoring/application.yaml`; the daily summary the operator ships to Loki is the history, see [History](). Two artifacts recur. A `ConfigAuditReport` has no TTL and goes only when its ReplicaSet is garbage-collected, so superseded revisions keep their findings on the dashboard ([aquasecurity/trivy-operator\#3069]()); when a count looks wrong, check whether the ReplicaSet in the report name still has replicas. `KSV-0125` ("untrusted registry") fires on nearly every container because the check's built-in list is Azure, ECR and GCR, which Trivy Operator does not expose as a parameter; it says nothing about the images. ## Filing policy Filing goes out under a real name, permanently, and most projects have said what they want: - **Argo CD** asks in `SECURITY.md` not to raise scanner findings. - **Home Assistant** (Open Home Foundation) does not accept issues or PRs from autonomous agents. - **Rook** accepts AI-assisted contributions with disclosure, submitted by a human. - **Routine Go churn is not a finding.** `grpc` or the stdlib one patch behind is fixed by the next release every project here takes automatically. "Waiting on release" means the fix is merged upstream and Renovate takes the release when it exists; nothing here speeds it up. ## What is load-bearing `ConfigAuditReport`, `RbacAssessmentReport` and `InfraAssessmentReport` are fixable here rather than upstream. Two checks are most of the volume: `KSV-0014` (read-only root filesystem) and `KSV-0118` (empty pod-level `securityContext`, whatever the containers set). | Where | Why it stays | | --- | --- | | Cilium, cilium-envoy, kube-vip, the kubeadm static pods, Rook OSDs and mons, node-exporter, Kured | Host network, host PID, privileged and added capabilities are what these do; `pod-security.yaml` enforces `privileged` in those namespaces for this reason | | `etcd-backup` and `etcd-defrag` CronJobs | Host network, because etcd listens on the node's loopback; root, because the client certificates are `600 root` | | Velero node agent | Root and capabilities stay: kopia reads every pod volume, whoever owns the files | | Images that start as root by design (Home Assistant) | `runAsNonRoot` or a read-only root would change how they run; seccomp is set | | Read-only root on charts that write under `/` at runtime (Grafana sidecars, OpenBao) | The chart or image decides where; guessing costs an outage (\#663) | | Every critical or high `ClusterRole` | A chart's operator role, or Kubernetes' own `admin`, `edit` and `cluster-admin`; not narrowable without forking a chart — see [Security Posture]() | | The kubelet unit at 644: `KCV-0069` | Accepted: it ships in the signed, read-only sysext and holds no secret. The kubelet config, the CA certificate and the CNI directory (`KCV-0077`, `KCV-0075`, `KCV-0056`) are set to the CIS modes by the `tmpfiles` rule in `ansible/templates/butane_node_config.yaml.j2`, because kubeadm rewrites them at 644 on every `init`, `join` and `upgrade`; the dispositions are in [Triage 2026-09-20]() | # Development # Development How to change this repository without surprising the cluster. Manifests live under `payload/` and are applied by ArgoCD; documentation lives under `docs/`; workloads live in [`homelab-apps`](). - **[Contributing]()** --- Repository setup, the checks to run, commit conventions, and how the documentation site is built and published. - **[Adding a Workload]()** --- Deploying a new application, from the ArgoCD Application through to the documentation page the build fails without. - **[Adding a Platform Component]()** --- A new directory under `payload/platform/`, and everything a component needs before its first sync: the Application, the policies, the pin and the page. - **[Maintenance]()** --- The CI workflows, and what Renovate is and is not allowed to merge on its own. # Adding a Platform Component A platform component is a directory under `payload/platform/` with exactly one `application.yaml`; the `platform` ApplicationSet turns it into an ArgoCD Application on the next poll. Workloads go in the [other repository](). Why the platform is shaped this way is in [GitOps Strategy](). ## Steps 1. **Create the directory** with a complete `application.yaml` copied from a sibling: the generator copies its labels, annotations, finalizers and `spec` verbatim, so a fragment is not an Application. It carries `project: infra`, like every component except the two that configure ArgoCD itself. Keep the [sync policy]() as it is — every component carries the same one, `retry` included, and a chart that lands before its CRDs stays failed without it. 2. **Pin the version once**, as `targetRevision` in that file. The `Makefile` reads the bootstrap versions out of these manifests and Renovate bumps them there — see [Version pins](). 3. **Put each resource with what it needs, not with what it configures.** Nothing orders the Applications, so a `ClusterSecretStore` sits with OpenBao and the issuers are their own component — see [Bootstrap convergence](). A resource whose CRD comes from another component gets `argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true`. A component with an `ExternalSecret` needs its namespace in both lists of `payload/platform/openbao/cluster-secret-store.yaml`, or the ExternalSecret reads `denied by spec.condition`. A chart from a new repository needs that host in the repo-server's policy in `payload/platform/security/network-policies/argocd.yaml`, or the Application reads `ComparisonError`. 4. **Label the namespace** in `payload/platform/security/pod-security.yaml` with the Pod Security Admission level it demonstrably needs, one sentence of why next to any `privileged`, and add a `privileged` one to the list in `namespace-privileged-allowed` — see [Security Policies](). Then **deny it to workloads**: add `"!"` to the `apps` project's destinations in `payload/platform/argocd-projects/projects.yaml`, which otherwise lets every workload Application write into it. A `privileged` namespace is the one that matters: the admission policy guards new Namespaces, not deployments into an existing one, so a workload commit could place a host-networked pod on every node. 5. **Write its network policy** as `payload/platform/security/network-policies/.yaml`: the `default-ingress` and `default-egress` deny pair plus one policy per workload that needs more, copied from the namespace most like it. New HTTP rules start as `http: [{}]` until the [rollout]() has recorded a week of requests. 6. **Expose a UI** with an `HTTPRoute` on `infra-gateway` next to the component, and put it behind Authentik — see [Gateway API]() and [Authentik](). A route that crosses namespaces needs a `ReferenceGrant` in the target namespace. 7. **Check Renovate sees the pin.** Chart versions are covered; an image tag inside `valuesObject` needs a `# renovate:` comment — see [Maintenance](). 8. **Write the page** under `docs/platform/` with the shared skeleton — intro, At a glance, Configuration, Usage, Health check, Pitfalls, Recovery only with a real runbook — add it to the nav in `zensical.toml` and to the [Components table](). The strict docs build fails on a page the nav does not list. 9. **Open the PR** and run the checks in [Contributing](). After the merge the Application appears within one polling interval; a Degraded one is usually waiting on a CRD, a Secret or OpenBao — see [Troubleshooting](). # Adding a Workload Workloads live in [`JanWelker/homelab-apps`](), not here. The `apps` ApplicationSet generates one ArgoCD Application per `*/application.yaml` in that repository, so adding an application is creating a directory there and pushing it. Why the split exists is in [GitOps Strategy](). ## What the platform already does for you | You want | You write | The platform does | | --- | --- | --- | | A hostname | An `HTTPRoute` with `hostnames: [app.k8s.wlkr.ch]` | [external-dns]() publishes the Route53 record | | HTTPS | Nothing | `apps-gateway` terminates TLS with a `*.k8s.wlkr.ch` wildcard | | Storage | A PVC with `storageClassName: rook-ceph-block` | [Rook-Ceph]() replicates it three ways, one copy per host | | A database | A CloudNativePG `Cluster` | [The operator]() generates credentials and runs failover | | A secret from outside the cluster | An `ExternalSecret` on the `openbao-apps` store, and an `external-secrets-vault` ServiceAccount — see [Adding a secret]() | [OpenBao]() holds it under `kv//`, the only prefix the namespace can reach; ESO renders a `Secret` | | Metrics | A `ServiceMonitor` or `PodMonitor` | Prometheus scrapes it | | Logs | Nothing | Alloy ships every container's stdout to Loki | | Backups | Nothing, for a PVC | Velero snapshots it nightly | ## The rules Restated in the workloads repository's `CONVENTIONS.md`, which is the copy to keep current. 1. **Exactly one `application.yaml` per directory**, a complete ArgoCD `Application`: the generator copies its labels, annotations, finalizers and `spec` through unchanged. 2. **`project: apps`.** Anything else fails the whole ApplicationSet. The project, in `payload/platform/argocd-projects/projects.yaml`, lists the chart repositories workloads may pull from; a new chart repository extends that list first, in this repository, and the repo-server's egress policy in `payload/platform/security/network-policies/argocd.yaml`, or the Application reads `ComparisonError`. 3. **The workload owns its namespace.** Ship a `namespace.yaml` at sync wave `-2` with the three Pod Security Admission labels, annotated `Prune=false`, because pruning a `Namespace` deletes every PVC inside it. A namespace created by `CreateNamespace=true` has no labels and enforces nothing. 4. **PostgreSQL is a CloudNativePG `Cluster`**, never a chart's bundled database — see [the contract](). 5. **Ship a `CiliumNetworkPolicy`** at sync wave `-2`, not a plain `NetworkPolicy`, which blocks health probes — see [Security Policies](). 6. **Official upstream sources only**: the vendor's chart or image, not a repackager's. 7. **Pin every version** and let Renovate move it. No `latest`. ## Step 1: Create the directory ```bash git clone git@github.com:JanWelker/homelab-apps.git cd homelab-apps mkdir my-app ``` ## Step 2: Write the Application `my-app/application.yaml`, for an app built from plain manifests: ```yaml --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app namespace: argocd annotations: argocd.argoproj.io/manifest-generate-paths: . spec: project: apps source: repoURL: https://github.com/JanWelker/homelab-apps.git targetRevision: HEAD path: my-app directory: exclude: "application.yaml" destination: server: https://kubernetes.default.svc namespace: my-app syncPolicy: automated: prune: true selfHeal: true retry: limit: 10 backoff: duration: 30s factor: 2 maxDuration: 5m syncOptions: # No CreateNamespace: namespace.yaml below creates it, with the Pod # Security labels an implicitly created one would not have. - ServerSideApply=true ``` For a Helm chart that also needs manifests of its own, use a `sources` list with the chart in one entry and this repository in another; `dependency-track/application.yaml` is the worked example. The `syncPolicy` is the one every platform Application carries, `retry` included; without it a workload that referenced a platform resource a minute too early stays failed — see [Sync policy](). ## Step 3: Add the manifests ```text my-app/ ├── application.yaml ├── namespace.yaml # Pod Security labels, sync wave -2 ├── database.yaml # CloudNativePG Cluster, sync wave -1, if it needs one ├── deployment.yaml ├── service.yaml ├── httproute.yaml # if you want it reachable └── networkpolicy.yaml # CiliumNetworkPolicy, sync wave -2 ``` ### The namespace ```yaml --- apiVersion: v1 kind: Namespace metadata: name: my-app labels: pod-security.kubernetes.io/enforce: baseline pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted annotations: # Pruning a Namespace deletes the PVCs inside it. argocd.argoproj.io/sync-options: Prune=false argocd.argoproj.io/sync-wave: "-2" ``` Set `enforce` to what the image demonstrably needs (`restricted` if it runs non-root, `baseline` if not) and leave `audit` and `warn` stricter so the gap stays visible. ### If it needs a database ```yaml --- apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: my-app-db namespace: my-app annotations: # Before the application: ArgoCD has a health check for this kind and # waits for "Cluster in healthy state" before starting the next wave. argocd.argoproj.io/sync-wave: "-1" spec: instances: 1 storage: size: 10Gi storageClass: rook-ceph-block bootstrap: initdb: database: my-app owner: my-app ``` That produces a Secret `my-app-db-app` holding `username`, `password`, `host`, `port`, `dbname` and a ready-assembled `uri`; point the application at those keys. The operator polls each instance on port `8000` from `cnpg-system`, so the network policy below must admit it. ### The network policy ```yaml --- apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: default-ingress namespace: my-app annotations: # With the namespace, ahead of the database at -1. argocd.argoproj.io/sync-wave: "-2" spec: endpointSelector: {} ingress: # Within the namespace, and the node for health probes. - fromEndpoints: - {} - fromEntities: - host - remote-node # The Gateway, on the container port behind the Service the HTTPRoute # targets: a policy matches the pod's port, not the Service's. - fromEntities: - ingress toPorts: - ports: - port: "8080" protocol: TCP # Only if a CloudNativePG Cluster is in this namespace. - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: cnpg-system toPorts: - ports: - port: "8000" protocol: TCP --- apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: default-egress namespace: my-app annotations: argocd.argoproj.io/sync-wave: "-2" spec: endpointSelector: {} egress: # Within the namespace. DNS is not listed: the cluster-wide policy in # payload/platform/security/network-policies/ sends every pod's # lookups through the proxy, which is what makes toFQDNs work. - toEndpoints: - {} # Every name the application dials, by name. Authentik counts: its # hostname resolves to the Gateway's own address, which is world. - toFQDNs: - matchName: auth.k8s.wlkr.ch toPorts: - ports: - port: "443" protocol: TCP --- # Only if a CloudNativePG Cluster is in this namespace: the instance # manager reports to the API server. apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: database namespace: my-app annotations: argocd.argoproj.io/sync-wave: "-2" spec: endpointSelector: matchLabels: cnpg.io/podRole: instance egress: - toEntities: - kube-apiserver ``` Wave `-2` is load-bearing: at the default wave the policy is applied after the `Cluster` at `-1`, which cannot go Healthy until the `cnpg-system` rule exists, so the sync parks at `-1` forever. Drop the `ingress` rule if the Gateway does not reach this workload directly (a namespace fronted by Authentik's outpost takes its traffic from the `authentik` namespace). A Prometheus rule belongs here only with a ServiceMonitor, on that port, with `rules.http`; the shape and the rollout are in [Security Policies](). ### Exposing it ```yaml --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app namespace: my-app spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: apps-gateway namespace: kube-system sectionName: https hostnames: - "my-app.k8s.wlkr.ch" rules: - backendRefs: - name: my-app port: 80 ``` DNS and TLS need nothing further — see [Gateway API](). ### Putting a workload behind Authentik - **It speaks OIDC:** the `oauth2provider` blueprint lives in the workload's directory; its client credentials come from a `PushSecret` in the same directory writing `kv//config` — see [Generated secrets]() — plus one projected-volume source in `payload/platform/authentik/application.yaml`; the discovery URI is `auth.k8s.wlkr.ch`. - **It does not:** add a `proxyprovider` on the embedded outpost, point the `HTTPRoute` at `authentik-server` in the `authentik` namespace, add the namespace to `payload/platform/authentik/referencegrant.yaml`, and allow the outpost to reach the pods in the `authentik-server` egress policy. Either way it is two pull requests, one here, because a workload must not be able to put itself behind or take itself out from behind the cluster's authentication layer. Details in [Authentik](). ## Step 4: Open the pull request ```bash git checkout -b feat/my-app git add my-app/ git commit -m "feat: add my-app" git push -u origin feat/my-app gh pr create --fill ``` Once it merges, the ApplicationSet picks the directory up within its requeue interval (`requeueAfterSeconds` in `payload/workloads/applicationset.yaml`), or immediately after **Refresh** on the `apps` ApplicationSet in the ArgoCD UI. ## Step 5: Document it The page goes in the workloads repository at `docs/my-app.md`, registered in its `zensical.toml`, which publishes to [homelab-apps.wlkr.ch](). Both sites build with `--strict`, so an unregistered page fails CI. Say what the application is for, which hostname it answers on, what it stores and where, and what surprised you while deploying it. ## Troubleshooting | Symptom | Cause and check | | --- | --- | | The Application was never generated | Only `*/application.yaml`, one level deep, is read, and a `spec.project` other than `apps` fails the whole set naming the file. Read the controller log (below) | | The Application exists but will not sync | It has exhausted its retries, on a fresh cluster usually against a platform resource that arrived later than the retry budget. `argocd app sync ` — see [Sync policy]() | | `Synced`, health `Unknown`, workload running fine | A wave is waiting on a resource that will never go Healthy; with a database it is the missing `cnpg-system` rule (`kubectl get cluster` says `1/1`, status reads `Instance Status Extraction Error: HTTP communication issue`). Confirm with `kubectl -n argocd get application my-app -o jsonpath='{.status.operationState.message}'` and the Hubble check below | | Rule is in Git, still stuck | The policy's sync wave is wrong: the applied list has no `CiliumNetworkPolicy` while `Cluster` reads `Running`. Fix the wave, then `argocd app terminate-op my-app`; a stuck operation does not pick up a new revision | | Pods will not start; events mention a security policy | `enforce` in `namespace.yaml` is stricter than the image needs. Loosen `enforce`, keep `audit` and `warn`, so violations stay in the [audit log]() — see [Pod Security Admission]() | | Login redirects in a loop, or every link is `http://` | Requests arrive from Cilium's Envoy inside the [pod subnet]() with TLS terminated at the Gateway. Set the application's trusted-proxy and "overwrite protocol" settings | ```bash # Why an Application was not generated kubectl -n argocd logs deploy/argocd-applicationset-controller | tail -50 # Drops from the operator, on the node running the database kubectl -n kube-system exec -c cilium-agent -- \ hubble observe --verdict DROPPED --from-namespace cnpg-system --last 20 # What the stuck sync has applied so far kubectl -n argocd get application my-app -o jsonpath=\ '{range .status.operationState.syncResult.resources[*]}{.kind}{"\t"}{.hookPhase}{"\n"}{end}' ``` # Contributing A merge to `main` is a deployment: ArgoCD applies `payload/` to real hardware, and there is no staging cluster in between. The checks below exist to make that routine. ## Setup ```bash uv sync # virtualenv and dependencies, including dev tools uv run pre-commit install # the same linters CI runs ``` ## Checks ```bash uv run pre-commit run --all-files # everything below, plus gitleaks ``` The same checks, one at a time: ```bash uv run yamllint -f github . uv run zensical build --clean --strict # docs site markdownlint-cli2 '**/*.md' '!**/.venv' '!.agent' uv run ruff check . && uv run ruff format --check . # python only uv run pytest -q # python only cd ansible && uv run ansible-lint # ansible only ``` Before touching `payload/`, render what Helm will actually apply; the `kubectl` line needs a kubeconfig, since even a client-side dry run asks the API server for its resource types. ```bash helm template / --version -f kubectl apply --dry-run=client -f payload/platform// ``` `--strict` is what CI uses. A page not registered in `nav` in `zensical.toml` fails the build, deliberately. On a pull request, the `argo-diff-preview` workflow comments the rendered ArgoCD manifest diff between `main` and your branch; read it, since a three-line values change can render as two hundred lines of different objects. The full list of workflows is in [Maintenance](). ## Make targets Every target in the `Makefile`, and the page that explains it. The ones that touch the cluster read `output/kubeconfig` (`KUBECONFIG=` picks another). | Target | Does | Page | | --- | --- | --- | | `setup` | Creates the virtualenv with every tool | [Setup](<#setup>) | | `download` | Fetches and verifies the Flatcar image, syslinux and the sysexts | [Quickstart]() | | `config` | Renders the Ignition configs, PXE menus and credentials from the inventory | [Quickstart](), [Ansible]() | | `artifacts` | `download` then `config` | [Quickstart]() | | `serve` | Runs the DHCP-proxy, TFTP and HTTP boot server | [Boot server]() | | `reinstall` | Arms a node (`LIMIT=`) or, after a typed confirmation, every node to wipe and install on its next network boot | [Node Lifecycle]() | | `reinstall-cancel` | Disarms them again | [Switching back to local boot]() | | `kubeconfig` | Copies the admin kubeconfig to `output/kubeconfig` | [Quickstart]() | | `check-context` | Guard the cluster targets depend on: fails without a kubeconfig | | | `bootstrap` | `install-cilium`, `install-argo`, `bootstrap-apps` in one go | [Quickstart]() | | `install-cilium` | Gateway API CRDs and Cilium, at the pinned versions | [Quickstart]() | | `install-argo` | The argo-cd chart, at the pinned version | [Quickstart]() | | `bootstrap-apps` | The AppProjects and the `argocd` Application; ArgoCD owns everything after | [Quickstart]() | | `storage-check` | Walks the Rook chain and binds a test PVC | [Is storage ready?]() | | `bao-init` | Initialises, unseals and configures OpenBao once | [OpenBao]() | | `bao-unseal` | Unseals every sealed replica from the key file (`KEYFILE=` picks another) | [OpenBao]() | | `bao-policies` | Rewrites ESO's policies and Kubernetes auth roles | [OpenBao]() | | `bao-secrets` | Prompts for the typed secrets and writes them | [OpenBao]() | | `bao-rotate-root` | Replaces a kept root token with the operator login | [OpenBao]() | | `claude-session-pat` | Stores (`SESSION=`) or removes (`DELETE=1`) one agent session's GitHub token | [Claude agents]() | | `untaint`, `taint` | Removes and restores the control-plane taint | [Single-node clusters]() | | `fonts`, `fonts-check` | Downloads the pinned webfonts, or diffs them against what is committed | [Fonts](<#fonts>) | | `clean-artifacts`, `clean` | Deletes the generated files, or everything under `output/` after asking | [Repository layout]() | ## Branches and commits Branch names are prefixed by area: `docs/`, `feat/`, `fix/`, `chore/`. Commit messages follow [Conventional Commits](): ```text docs: explain the stack choices and collect the known limitations fix(bootstrap): repoint the install targets at the files that exist chore(deps): update helm release kube-prometheus-stack to v89 ``` The body explains *why*; the diff already says what. > [!WARNING] > > **Do not hand-edit version numbers** > > `targetRevision` in the manifests and the versions in `ansible/inventory.yaml` are owned by Renovate — see [Maintenance](). A hand bump conflicts with, or is reverted by, the next Renovate PR. The bootstrap targets in the `Makefile` read the same `targetRevision` values rather than pinning their own — see [Version pins](). ## Review `.github/CODEOWNERS` assigns every path to the repository owner, which requests that review; it is not required, because an author cannot approve their own pull request. What `main` requires is the `ci-ok` check; which Renovate PRs merge on it alone is in [Automerge policy](). `.agent/rules/general-rules.md` holds the standing rules for AI coding agents working here: GitOps only, docs in `docs/`, no hand-edited versions. Keep it in sync when those conventions change. ## Documentation The site is built with [Zensical]() from `docs/`, configured in `zensical.toml`, and published to GitHub Pages by `.github/workflows/docs.yaml` at [https://homelab.wlkr.ch/](). It is deliberately not hosted on the cluster, so it stays readable when the cluster is not. The build also writes `llms.txt`, `llms-full.txt` and a `.md` twin of every page for LLM clients ([llmstxt.org]()); the sections of `llms.txt` are listed in `zensical.toml` under `plugins.llmstxt`, so a new section in `nav` needs an entry there too. | Stage | What happens | | --- | --- | | Build | On push to `main` touching `docs/**`, `overrides/**`, `zensical.toml`, `pyproject.toml` or `uv.lock`, `docs.yaml` runs `zensical build --clean --strict` | | Deploy | The workflow drops a `.nojekyll` marker into `site/` (otherwise Pages runs Jekyll, which drops paths it considers private) and pushes to `gh-pages`, leaving `pr-preview/` untouched | | Previews | `preview.yaml` builds every pull request matching the same paths, forks included, and publishes it under `pr-preview/` on `gh-pages`; closing the PR removes it. Its path filter must match `docs.yaml`, because this is the only `--strict` build a PR gets. Only the publish step is skipped for fork PRs: they have no write access | Locally: ```bash uv run zensical serve # http://localhost:8000, rebuilds on save uv run zensical build # one-off build into site/ (gitignored) ``` ### Conventions - Every page is registered in `nav`; cross-references are relative Markdown paths (`platform/openbao.md`, `../quickstart.md`) so they resolve on the site and on GitHub. - Diagrams use Mermaid fences; Zensical loads the runtime on pages that contain one. - Markdown is linted by `lint-markdown.yaml` and the `markdownlint-cli2` pre-commit hook. `MD046`: a blank line followed by an indented line reads as a code block, so keep `!!!` admonition bodies to a single paragraph. `MD007`: nested lists indent by four spaces. - Inline HTML is limited to `
` (`MD033` in `.markdownlint-cli2.yaml`), which the card grids on hub pages need. - Theme templates are overridden by a same-named file under `overrides/` (`theme.custom_dir`). Currently only `partials/source.html`, which drops the repository-facts call that 404s because this repository publishes no releases. ### Fonts Fonts are self-hosted so the site makes no third-party requests: `theme.font = false` in `zensical.toml`, `docs/stylesheets/fonts.css` declares Inter and JetBrains Mono from the `woff2` files in `docs/assets/fonts/`, and each SIL OFL licence sits next to them. `scripts/update-fonts.sh` pins both upstream releases ([Inter](), [JetBrains Mono]()) and Renovate tracks the pins. > [!WARNING] > > **A font-pin PR needs a second commit** > > Renovate can move the pin but cannot write the binaries, so its PR changes one line and the `woff2` files are still the old release. Check out the branch, run `make fonts` to download what the pin now names, and commit the result. `make fonts-check` (run by `fonts-check.yaml` on every such PR) diffs the downloads against what is committed and fails the branch until that commit arrives, which is what holds the automerge. # Maintenance Two things keep the repository from rotting: GitHub Actions workflows that lint everything, and Renovate, which chases dependencies. ## CI workflows All in `.github/workflows/`. | Workflow | Trigger | What it checks | | --- | --- | --- | | `lint-ansible.yaml` | `ansible/**` | `ansible-lint` | | `lint-python.yaml` | `boot_server/*.py`, `scripts/*.py`, `tests/*.py`, `pyproject.toml`, `uv.lock` | `ruff check`, `ruff format --check` and `pytest` on the boot server and the scripts | | `lint-yaml.yaml` | `**/*.yaml` | `yamllint` | | `lint-markdown.yaml` | `**/*.md` | `markdownlint-cli2` | | `docs.yaml` | push to `main` under `docs/**`, `overrides/**`, `zensical.toml`, `pyproject.toml`, `uv.lock` | Builds the site with `--strict` and publishes it — see [Contributing](). The build runs with a read-only token; only the deploy job, which runs no dependency code, can push | | `preview.yaml` | pull requests on the same paths | Publishes a preview under `pr-preview/` from the build job's artifact, same split as `docs.yaml`; fork PRs skipped | | `argo-diff-preview.yaml` | pull requests under `payload/**` | Comments the rendered ArgoCD manifest diff against `main`; fork PRs skipped | | `image-scan.yaml` | pull requests under `payload/**` | `scripts/image-scan-diff.py`: renders every Application on `main` and on the PR, scans with Trivy only the images that changed, and fails on a fixable CRITICAL the replaced image did not carry. A finding already [waiting on a release]() does not block the bump that gets closer to it | | `renovate-validate.yaml` | `renovate.json` | `renovate-config-validator` | | `secret-scan.yaml` | every pull request | `gitleaks` over the PR's commits, the same version as the pre-commit hook; a manual run scans the whole history | | `ci-ok.yaml` | every pull request | Waits for every other check on the head commit and fails if one failed. The only check the ruleset on `main` requires, so the others keep their path filters | | `fonts-check.yaml` | `scripts/update-fonts.sh`, `docs/assets/fonts/**` | `make fonts-check`: the committed `woff2` files match the pinned releases — see [Fonts]() | ## Renovate Configured in `renovate.json`, on top of the shared preset [JanWelker/renovate-config](), which holds the policy for every repository. Every component gets its own pull request so a failing update never holds up an unrelated one; the only groups are the sets below that must move together. ### Automerge policy - **The preset's rules:** three days between a release and its PR, patch and minor automerge once `ci-ok` is green, majors open at 19:00 and wait for a human, and security fixes skip both the wait and the schedule. The preset's README says why for each. - **One exception: a Kubernetes minor waits for a human.** A Flatcar minor lands the same way a Grafana chart patch does, but `kubernetes_version` is walked forward by hand in version-skew order, and a node provisioned from an automerged minor would run a kubelet ahead of the API server and fail its `kubeadm join` — see [Upgrades](). - **Pairs that must move together are grouped into one PR:** `prometheus-operator-crds` with `kube-prometheus-stack`, because the CRDs must not lag the operator; `rook-ceph` with `rook-ceph-cluster`, because Rook requires both charts on the same version; and `markdownlint-cli2` in the workflow with its pre-commit hook, because the two must lint alike; `gitleaks` likewise; the `external-snapshotter` tag with the `snapshot-controller` image it pins, because upstream's manifest at a tag names the previous release's image (the image is a `registry.k8s.io` one, so the pair waits for a human like every third-party image there); the two ARC charts, because the controller deletes a scale set whose chart version differs from its own; and every Python pin (`requires-python`, `.python-version`, the `python` images), because `uv sync` fails when they name different minors. - **Python ranges bump.** `pyproject.toml` declares `>=` ranges, which a new release already satisfies; `rangeStrategy: bump` is what makes Renovate open a PR for the linters and the docs generator at all. - **Flatcar and Kubernetes bumps change what a newly provisioned node installs, not what a running node runs.** kubeadm cannot skip a minor, so a cluster that sits while two Kubernetes minors merge has to be walked forward one at a time — see [Upgrades](). - **A font bump cannot automerge without its second commit:** `fonts-check.yaml` fails on it and `ci-ok` waits for it — see [Fonts](). - `config:best-practices` pins GitHub Actions to commit SHAs and container images to digests, and collects every pin into one shared `renovate/pin-dependencies` branch. ### Scope Python (`pyproject.toml`, `uv.lock`), Docker images, GitHub Actions, Kubernetes and ArgoCD manifests, Helm values files (`payload/**/values.yaml`) and pre-commit hooks, plus custom regex managers for the versions in `ansible/inventory.yaml`, the font pins in `scripts/update-fonts.sh`, any `# renovate:` annotation and every CloudNativePG `imageName` under `payload/`, and the tool versions the workflows under `.github/workflows/` install by name rather than by action (the diff-preview image, `markdownlint-cli2`, `renovate` and Trivy). The `Makefile` is deliberately not tracked: the bootstrap targets read `targetRevision` out of the owning `application.yaml` — see [Version pins](). ### Manager rules A misconfigured manager fails silently: the run succeeds and nothing is checked. Prove a manager works by reading what it extracts: ```bash LOG_LEVEL=debug npx --yes renovate@latest --platform=local --dry-run=lookup ``` It needs no credentials for a public repository and prints every dependency it found and why it skipped any. The rules, each of which is written into `renovate.json`: - **`pre-commit` must be enabled explicitly** (`"pre-commit": {"enabled": true}`); Renovate ships that manager disabled, and a `packageRules` entry for it does nothing on its own. - **The `helm-values` manager reads only real values files.** An image tag inline in an Application's `helm.valuesObject:` is invisible to it; put a `# renovate: datasource=... depName=...` comment on the line above so the annotation manager picks it up, or move the values into a `values.yaml`. - **A custom manager that pins a full image reference needs `autoReplaceStringTemplate`.** Without it the pin cannot be written, and one unwritable pin fails the whole shared pin branch (`Error updating branch: update failure` on the Dependency Dashboard, the only symptom, since version bumps keep working). Capturing `currentDigest` only lets Renovate update a digest that is already there. - **A bare Helm `tag:` value goes in the `pinDigests: false` rule.** A tag like `3.4.4-alpine` has no position for a digest; leave one out and the same banner comes back. - **A two-part version needs a `versioningTemplate`.** The default `semver-coerced` rejects `6.03`, so the dependency is extracted and then dropped before lookup. `syslinux` and the Inter font pin carry a `regex:` versioning for this reason; Inter's also keeps `v4.0-beta9h`-style prereleases out. - **Match the registry's spelling, not the GitHub org's.** The diff-preview image is `dagandersen/argocd-diff-preview` on Docker Hub; a regex on `dag-andersen` matches nothing and reports nothing. - **Hold a major with a regex, not a range.** `"allowedVersions": "/^17\\./"` is matched before version parsing; a range like `<18` is graded by npm semver, where a three-part tag such as `17.11.1-trixie` parses as a prerelease and is dropped from every range that does not name one. Python is held at 3.14 this way until ansible-core supports 3.15. - **Check that the currently pinned version passes its own rule.** A filter that excludes what is deployed excludes everything, silently. ## Language statistics GitHub's language bar comes from [Linguist](), which counts bytes of files it classifies as a *language*. YAML is classified as data and Markdown as prose, so both are excluded by default and the bar reported this repository as Shell and Jinja. `.gitattributes` marks YAML detectable, and marks the three vendored Grafana dashboards as vendored, because their upstream JSON blobs are more than half the repository's YAML bytes and would otherwise be most of the bar. A vendored dashboard added later needs its own line there; GitHub recalculates on the next push to `main`.