Security Posture¶

This is a homelab on a private network, and several deliberate shortcuts follow from that. They are listed so that someone reading the manifests can tell a decision from an oversight.

Assumed trust boundary¶

The cluster assumes a trusted L2 network segment: anything with a port on it is treated as friendly. There is no VPN requirement, no mutual TLS between components, and no network segmentation inside the cluster.

The published hostnames are a partial exception. Certificates come from Let's Encrypt through a DNS-01 challenge against a public zone, so argo.infra.k8s.wlkr.ch and its siblings are publicly resolvable and appear in Certificate Transparency logs, even though they point at RFC1918 addresses. Plan names accordingly.

Provisioning¶

Provisioning is the least protected phase, by design: it has to work before any of the cluster's own security exists.

The mitigation is time: make serve is a foreground command, the bootstrap token has a 24 hour TTL and the certificate key expires after two hours. The encryption key has no clock, which is why the boot server listens only on boot_server_ip, serves files by name and lists nothing. Stop the boot server when provisioning is finished. Flatcar is installed to disk, so nothing needs it after a build — see Every boot after the first.

output/credentials/ holds the generated bootstrap token and certificate key in plaintext. The directory is 0700 and output/ is gitignored, but the values are reused across make config runs — the Ansible password lookup reads back an existing file. Delete them to force new ones.

Secrets¶

Secrets live in OpenBao and reach workloads as native Kubernetes Secret objects through the External Secrets Operator. Nothing sensitive is committed to Git.

• OpenBao is sealed with Shamir and unsealed by hand, so the 5 key shares are the root of trust for every other secret; losing all of them loses everything, and nothing outside the cluster holds a copy.
• A Kubernetes Secret is base64, not encryption. Anyone with get secrets in a namespace can read what ESO materialised there. Encryption at rest protects the bytes in etcd, not the API.

Encryption at rest¶

The API server encrypts secrets with secretbox before they reach etcd (ansible/templates/kubeadm.yaml.j2; the key file in ansible/templates/butane_node_config.yaml.j2). Without it an etcd backup, a stolen disk, or read access to /var/lib/etcd yields every credential the cluster holds.

With identity last, new writes are encrypted and Secrets written before it was enabled stay readable but are not rewritten. To encrypt what already exists, once the API servers have restarted:

kubectl get secrets -A -o json | kubectl replace -f -

The key is static, with no rotation, and lives in output/credentials/ beside the kubeadm token and certificate key — that directory is what decrypts etcd. A KMS provider would remove the static key at the cost of a dependency the cluster must reach before it can serve Secrets.

Runtime¶

Tetragon records every exec in every pod and alerts when anything but the control plane reads the cluster's key material. It is detection, not prevention: nothing is killed, and a policy that fails to load on a new kernel watches nothing until its alert fires.

Audit logging¶

The API server records who did what, to which object, and whether it was allowed; the audit half of the Pod Security Admission labels has nowhere to go without it. Policy, retention and queries are in Audit Logging.

Authorization¶

AppProjects constrain the workloads, not the platform. payload/platform/argocd-projects/projects.yaml defines three. infra allows sourceRepos: "*" and every group and kind in every namespace: it is this repository deploying this repository, and a restriction there guards against nothing the PR review does not. system is confined to the argocd namespace. apps has the boundary to draw, because its Applications come from a second repository with a lighter review: it accepts only the workloads repository and the chart repositories the workloads use, may not write into any platform namespace (authentik excepted, for the blueprint ConfigMap a workload ships there), and at cluster scope may create nothing but namespaces.

Network policy covers ten namespaces. openbao, cert-manager, external-secrets, monitoring, external-dns, kubelet-csr-approver, kured, logging, cnpg-system and trivy-system have default-deny ingress CiliumNetworkPolicy rules; every other namespace, and all egress everywhere, is unrestricted. See Security Policies.

Pod Security Admission is on, but mostly auditing. Every platform namespace carries enforce at the level it demonstrably needs and warn/audit at a stricter one, so violations are visible without breaking what runs: measure first, enforce second.

Image references are checked at admission. An untagged or latest image is refused; a registry outside the allowlist is recorded and warned about but admitted. Signatures are not verified: few of the upstreams sign, so a verifier would exempt nearly everything. See Security Policies.

Both Gateways admit routes from every namespace (allowedRoutes.namespaces.from: All). Any namespace can attach an HTTPRoute to infra-gateway and claim a hostname under *.infra.k8s.wlkr.ch.

Exposed interfaces¶

Every platform UI on the infra gateway is behind Authentik:

Authentik is therefore a dependency of reaching any of them; the break-glass paths are in When Authentik is down. OpenBao is left out deliberately: it holds Authentik's own database password, so putting it behind Authentik would be a loop.

What would tighten this up¶

Roughly in order of value against effort:

1. Extend default-deny ingress to the namespaces not yet covered, then start on egress — the larger and more breakable half.
2. Narrow sourceRepos on the AppProjects to this repository and the Helm repositories actually in use.
3. Restrict allowedRoutes on infra-gateway to the platform namespaces, and promote the registry allowlist to Deny.
4. Add a second Alertmanager receiver on a different transport, so a failure of the mail path is not itself unmonitored.
5. Move etcd encryption to a KMS provider, removing the static encryption_key in output/credentials/.
6. Get the backups out of the cluster: Velero and the etcd snapshots write to the Ceph object store they are backing up.

See Known Limitations for the operational counterparts.