# Authentik

[Authentik](<https://goauthentik.io/>) is the cluster's identity provider. Every platform UI sits behind one login instead of a chart-default password each — and Hubble UI, which ships no authentication at all, would otherwise publish every network flow in the cluster to anyone who could reach the hostname.

| Service | Authentication |
| --- | --- |
| ArgoCD | OIDC, local admin **disabled** |
| Grafana | OIDC, login form disabled |
| Home Assistant | Proxy outpost **in front of** its own login — see [Pitfalls](<#pitfalls>) |
| Umami | Proxy outpost **in front of** its own login; the tracker paths pass through — see [Pitfalls](<#pitfalls>) |
| Rook dashboard, Hubble UI, Prometheus, Alertmanager, Flowscape, Extractor admin UI | Proxy outpost |
| Wollbi-Fescht, Wollbi Adventsfenster | Proxy outpost on the admin paths, with no `infra-users` binding, see [Groups and roles](<#groups-and-roles>) |

## At a glance

|  |  |
| --- | --- |
| Namespace | `authentik` |
| Depends on | [OpenBao](<https://homelab.wlkr.ch/platform/openbao/index.md>) for its OIDC client secrets, [CloudNativePG](<https://homelab.wlkr.ch/platform/cloudnative-pg/index.md>) for the database, [Rook-Ceph](<https://homelab.wlkr.ch/platform/rook-ceph/index.md>) under it |
| If it is down | Every platform UI loses its only login — see [When Authentik is down](<#when-authentik-is-down>) |
| Health check | `kubectl -n authentik get pods,clusters` → server and worker Ready, `authentik-db` healthy |
| UI | `auth.k8s.wlkr.ch` |
| Files | `payload/platform/authentik/` |

## Configuration

### Two integration styles

**OIDC**, for applications that can do it themselves: ArgoCD and Grafana map group membership to a role. **Proxy**, for the rest: the hostname resolves to Authentik's embedded outpost, a reverse proxy that authenticates the request and only then forwards it to the backend. It needs nothing from the application and protects exactly as far as nothing else can reach the backend.

```
flowchart LR
    U([Browser]) --> GW[infra-gateway / apps-gateway]
    GW -->|argo, monitoring, OIDC workloads| APP[ArgoCD / Grafana / OIDC workloads]
    APP -.->|OIDC redirect to auth.k8s.wlkr.ch| AK[Authentik]
    GW -->|hubble, rook, prometheus, alertmanager, home, flowscape, analytics, extractor-admin| AK
    AK -->|authenticated| BE[Hubble UI / Ceph dashboard / ...]
```

### One hostname

Authentik answers on `auth.k8s.wlkr.ch` only, on `apps-gateway`, which the LAN reaches through the router's local record and the outside through Route53. Every OIDC client and the embedded outpost use that one name.

The outpost is why there is only one. A proxied application's route points at `authentik-server`, but the login itself is an OAuth dance: the outpost sends the browser to the outpost's `authentik_host` to authenticate and gets it back on the application's own hostname. With Authentik on two names and `authentik_host` on the LAN-only one, an application on `apps-gateway` worked at home and redirected outside users to a name that does not resolve. `blueprints.yaml` pins `authentik_host` because the embedded outpost otherwise fills it from `web.base_url` on first start and never updates it.

> [!WARNING]
>
> **A client must never mix names**
>
> The `iss` claim is the hostname the token was issued through. A discovery URI on one name and a redirect on another fails every login with an error that blames the token. Should a second hostname ever come back, pin each client to exactly one.

### Where the proxied routes point

A proxied application's route lives with the application: Hubble's in `payload/platform/cilium/`, Rook's in `payload/platform/rook-ceph/`, Prometheus's and Alertmanager's in `payload/platform/monitoring/`, and Home Assistant's, Flowscape's, Umami's, Wollbi-Fescht's, Wollbi Adventsfenster's and the Extractor admin UI's in the [workloads repository](<https://homelab.wlkr.ch/development/add-workload/index.md>), each with `authentik-server` as `backendRef`. Gateway API forbids a cross-namespace `backendRef` unless the target namespace grants it, so `referencegrant.yaml` here allows `HTTPRoute` objects from exactly those namespaces to that Service only. A proxied workload is therefore two pull requests, by design: a workload cannot take itself out from behind the authentication layer on its own.

### Database

`database.yaml` is a CloudNativePG `Cluster` with replicas, the only one on the cluster that has any. Every login on every platform UI goes through Authentik, so its database must survive a node drain: with one instance a reboot kills the database; with replicas, one takes over within a minute, once `enablePDB: false` lets the drain evict the primary. The chart's own Postgres subchart is off — both rules are in [the contract](<https://homelab.wlkr.ch/platform/cloudnative-pg/#the-contract>).

The replicas only survive a node loss while they sit on separate nodes, which the operator's own `preferred` anti-affinity does not guarantee — hence the `topologySpreadConstraints` block, under the rule in [Replica placement](<https://homelab.wlkr.ch/platform/#replica-placement>).

### Chart values

| Setting | Why |
| --- | --- |
| `authentik.disable_update_check`, `authentik.disable_startup_analytics` and `authentik.avatars` | Telemetry is switched off in config, not permitted by network policy — the same rule as Grafana, Loki and Alloy. `avatars` only seeds a fresh database: since 2024.2 the setting lives in **System → Settings**, and an existing install keeps calling Gravatar until it is set to `initials` there. Blueprints cannot set it: system settings are an internally managed model, and upstream declined a config-level override in [goauthentik/authentik\#26470](<https://github.com/goauthentik/authentik/issues/26470>) |
| `authentik.web.base_url` | Authentik builds e-mail links and outpost redirects from it and cannot infer it; unset, every admin page shows "The base URL has not been configured". The chart value backfills the tenant, so a rebuilt cluster needs no click |
| `metrics.enabled` and `metrics.serviceMonitor.enabled` | The ServiceMonitor renders only when both are set; the switch alone produces nothing, silently. The worker is scraped too: tasks, outpost state and blueprint runs are measured there |
| `postgresql.enabled: false`, `authentik.postgresql.host: authentik-db-rw` and the `global.env` entry | The database is the `Cluster` above. The password is the one CloudNativePG generated into `authentik-db-app`, read as `AUTHENTIK_POSTGRESQL__PASSWORD` from that Secret; it is never copied into OpenBao or Git |
| `server.topologySpreadConstraints` | The chart's `podAntiAffinity` preset is `soft`, which is not enough to hold the replicas apart — see [Replica placement](<https://homelab.wlkr.ch/platform/#replica-placement>). The worker is not replicated and needs none |
| `worker.podAnnotations` `homelab.wlkr.ch/secret-generation` | Bumped whenever `authentik-secrets` or a workload's `authentik-secrets-<workload>` gains a key, so ArgoCD restarts the worker in the same sync — see [Pitfalls](<#pitfalls>) |
| Workload blueprints as a `projected` volume, `optional: true` | `blueprints.configMaps` renders a plain `configMap` volume, and the kubelet refuses a pod whose ConfigMap is missing. Workload blueprints arrive with the [workloads](<https://homelab.wlkr.ch/architecture/gitops/#workloads-live-in-a-second-repository>), which a fresh cluster may not have yet, so a required mount would keep the worker from starting |

### Where a blueprint lives

Providers and applications are blueprints, so a rebuild reproduces them. `!Find` resolves objects Authentik ships, `!KeyOf` references another entry in the same blueprint, and `!Env` reads an environment variable, which keeps client secrets out of Git. `blueprints_discovery` runs on worker startup, hourly and on a file watcher, asynchronously, so anything configuring itself against a provider must tolerate it not existing yet.

| What | Lives in | Why there |
| --- | --- | --- |
| Platform providers, applications, the outpost | `blueprints.yaml` | Mounted into the worker as a ConfigMap |
| A workload's provider and application | The workload's directory in the workloads repository, as the ConfigMap `authentik-blueprint-<application>` targeted at `authentik` | The provider and the application it authenticates change in one commit; the [`authentik-foreign-objects`](<https://homelab.wlkr.ch/platform/security-policies/#admission-policies>) policy refuses any other object a workload sends here |
| Client credentials | Generated by a `PushSecret` beside their reader — `secrets.yaml` for the platform's, the workload's own `secrets.yaml` for a workload's — and read here by `secrets-<workload>.yaml` | A workload can mint a pair but not register it: the `ExternalSecret` and mount entry here keep SSO onboarding a deliberate two-repository act |
| The mount entry and `secret-generation` | `application.yaml` | One projected-volume source per workload, with or without credentials; a new credential needs the worker restarted |
| The outpost's `providers:` list | `blueprints.yaml` | It replaces one global object; two repositories writing it would overwrite each other |
| `referencegrant.yaml` | Here | A proxied workload's route needs granting from this side |

### One issuer for OpenCloud

OpenCloud's web, desktop, iOS and Android clients are four OIDC clients with IDs fixed by the clients, so four providers in Authentik, each under an application of its own. The server and the native clients verify every token against one issuer, and Authentik derives a provider's issuer from its application slug, so the four would disagree. All four therefore use `issuer_mode: global`, whose issuer is the bare hostname, and the `HTTPRoute` rewrites `/.well-known/openid-configuration` on that hostname to the `opencloud` application's discovery document, which is where OpenCloud and its clients fetch it. The document names the global issuer and the shared endpoints, so it serves the other three providers as well. Nothing else uses the root path: every other provider is found under its slug.

### User settings flow

Dependency-Track keys its accounts on the `preferred_username` claim, so a user who could rename themselves could take over another's account. `05-user-settings.yaml` therefore gives the default brand its own user settings flow with no username field; a username changes only under **Directory → Users**. It replaces the brand's flow instead of editing the default one because the default blueprint rewrites its own objects on every upgrade, and because the **System → Settings** switch `default_user_change_username` is not a model a blueprint can set.

### Authentication flow

`auth.k8s.wlkr.ch` is reachable from the Internet, and the default authentication flow lets a user without a device in on a password alone. `06-authentication.yaml` gives the default brand its own flow out of the default identification, password and login stages, with two additions. It is an own flow for the same reason as the [user settings flow](<#user-settings-flow>): the default blueprint rewrites its own objects on every upgrade.

| Addition | Why |
| --- | --- |
| Authenticator validation with `not_configured_action: configure`; TOTP and WebAuthn as setup stages, static tokens accepted as a device class | A password alone is one leak away from every application, so a user without a device enrols one at the next login instead of being let through. Static tokens are the recovery codes for a lost phone, enrolled afterwards under **Settings → MFA devices**, which is why they are not offered as the first device |
| A deny stage after identification, selected by a reputation policy on the address and the username with threshold `-5` | Every failed login scores both `-1`, so after five an address or a username is refused for a day (`AUTHENTIK_REPUTATION__EXPIRY`); the password stage's own five-attempt cancel only ends the current attempt. The policy *passes* for a bad score, so it selects a deny stage rather than guarding the flow, and it is evaluated when the stage is reached because the flow's own policies see an anonymous request before the username is known |

To lift a lockout early, delete the score under **Customization → Policies → Reputation scores**.

### Bootstrap credentials

Authentik's built-in bootstrap blueprint creates `akadmin` from `AUTHENTIK_BOOTSTRAP_PASSWORD` only while no `akadmin` exists, and reads nothing from it afterwards. `06-authentication.yaml` then sets the account `is_active: false`: a password-only superuser that never logs in is the one worth guessing, and a disabled one authenticates nowhere. The blueprint touches only `is_active`, so the password and the `authentik Admins` membership stay for break-glass.

Break-glass is `ak shell` from a cluster-admin kubeconfig, which nothing in Authentik can lock out:

```bash
kubectl -n authentik exec deploy/authentik-server -c server -- ak shell -c \
  'from authentik.core.models import User; User.objects.filter(username="akadmin").update(is_active=True)'
```

Then log in with `bootstrap-password` from OpenBao — see [Generated secrets](<https://homelab.wlkr.ch/platform/openbao/#generated-secrets>) — enrol a second factor like any user, and switch the account off again with the same command and `is_active=False`: a blueprint is re-applied only when its file changes, so nothing switches it off for you.

| Variable | Set | Why |
| --- | --- | --- |
| `AUTHENTIK_BOOTSTRAP_PASSWORD` | Always, from `bootstrap-password` | Without it a fresh install creates `akadmin` with no usable password, and the initial-setup flow that sets one is open to whoever reaches `auth.k8s.wlkr.ch` first. Blanking it after the first start buys nothing: the account keeps the password |
| `AUTHENTIK_BOOTSTRAP_TOKEN` | Never | It mints a non-expiring superuser API token, `authentik-bootstrap-token`, which nothing here uses, and while the variable is set the blueprint recreates the token whenever it is deleted |

A cluster installed before the token was dropped still holds it: delete `authentik-bootstrap-token` under **Directory → Tokens and App passwords**, once the worker has restarted without the variable.

### Groups and roles

Authorisation is group membership. The blueprints create these groups; membership is set by hand under **Directory → Groups**:

| Group | Grants |
| --- | --- |
| `infra-users` | Past the Authentik login of every infra application: ArgoCD, Grafana, Hubble, the Ceph dashboard, Prometheus, Alertmanager, Flowscape and the Extractor admin UI. Grafana `Viewer` for a direct member |
| `argocd-admins` | ArgoCD `role:admin`, and `infra-users` |
| `argocd-viewers` | ArgoCD `role:readonly`, and `infra-users` |
| `grafana-admins` | Grafana `Admin`, and `infra-users` |
| `grafana-editors` | Grafana `Editor`, and `infra-users` |
| `wollbi-editors` | A Payload editor on advent.wollbi.ch and fest.wollbi.ch, created on first sign-in; declared in the two sites' blueprints in the workloads repository |

Every infra application carries a policy binding to `infra-users`, so an account outside it, such as a wollbi editor's, is refused by Authentik before the application sees it. The four role groups have `infra-users` as their parent, and Authentik counts a child group's members as members of the parent. Grafana only sees the groups a user belongs to directly, and with `role_attribute_strict` it refuses a login that matches none of its three. ArgoCD's `policy.default` is empty, so a user in neither ArgoCD group gets **no** access. The wollbi sites bind `wollbi-editors` instead. Home Assistant and Umami have no binding: they have logins of their own.

## Usage

### Client secrets

The OIDC client IDs and secrets are generated once into OpenBao and read by both sides — Authentik through `!Env`, ArgoCD and Grafana through their own `ExternalSecret` — so a rebuilt Authentik gets the same credentials and nothing is copied out of a UI. The `PushSecret` in `secrets.yaml` writes `kv/authentik/config` once — see [Generated secrets](<https://homelab.wlkr.ch/platform/openbao/#generated-secrets>) — with the secret key, the bootstrap password and the ArgoCD and Grafana client pairs; its `Password` generator's `secretKeys` is the list of keys. A workload's client credentials go in the workload's own path (Dependency-Track's in `kv/dependency-track/config`, each read by its own `secrets-<workload>.yaml`, mounted `optional`): rewriting `kv/authentik/config` would rotate the secret key that signs every session and token, and External Secrets fails an `ExternalSecret` whole when one key is missing, so a shared one would leave a cluster without the workload with no `AUTHENTIK_SECRET_KEY` either.

The first administrator is created through `akadmin`, switched on for the occasion as in [Bootstrap credentials](<#bootstrap-credentials>); the steps are in [Quickstart step 12](<https://homelab.wlkr.ch/quickstart/#setup>).

### Adding an application

1. Generate its client credentials into OpenBao with a `PushSecret` beside the workload's own `ExternalSecret`, and read them here with an `ExternalSecret` mounted `optional`, like `secrets-dependency-track.yaml`. A public client (PKCE, as OpenCloud's web and native clients) has no credentials: its client IDs are fixed by the client and sit in the blueprint in clear, so this step and step 3 do not apply.
2. Mount the workload's blueprint: in `application.yaml`, a `configMap` source named `authentik-blueprint-<application>` with `optional: true` under `worker.volumes`, in the `projected.sources` of `blueprints-workloads`.
3. Hand the worker the credentials: in the same file, a `secretRef` to `authentik-secrets-<workload>` with `optional: true` under `global.envFrom`, and bump `homelab.wlkr.ch/secret-generation`.
4. Add the provider and application blueprint: an OAuth2 provider with `grant_types`, or a proxy provider added to the outpost's `providers:` list.
5. For a proxied application, point its `HTTPRoute` at `authentik-server`, add its namespace to `referencegrant.yaml`, and add a `toEndpoints` rule for its pods to the `authentik-server` policy in `payload/platform/security/network-policies/authentik.yaml`: the outpost dials the backend, and without the rule the login succeeds and the request after it times out.
6. Bind `infra-users` to an infra application in the blueprint, as an `authentik_policies.policybinding` with the application as `target`. An application with no binding is open to every Authentik account.

### Adding a user

1. Create the account under **Directory → Users**.
2. Add it to `infra-users`, or to a role group from [Groups and roles](<#groups-and-roles>), under **Directory → Groups**. Membership is not blueprinted because that would put a username in the repository, and only a superuser passes the bindings without it.

## Health check

When every login fails, in this order:

1. Server and worker are Ready and the database is healthy:

   ```bash
   kubectl -n authentik get pods,clusters
   ```
2. The discovery document names the one hostname as `issuer`:

   ```bash
   curl -s https://auth.k8s.wlkr.ch/application/o/argocd/.well-known/openid-configuration | jq .issuer
   ```
3. The embedded outpost is healthy (`Outposts` in the admin UI, or):

   ```bash
   kubectl -n authentik logs deploy/authentik-server | grep -i outpost | tail
   ```
4. The worker applied the blueprints — a failed one names its file:

   ```bash
   kubectl -n authentik logs deploy/authentik-worker | grep -i blueprint | tail
   ```

## Pitfalls

> [!CAUTION]
>
> **A new client credential needs the worker restarted**
>
> `envFrom` injects `authentik-secrets` **once, when the pod starts**. A blueprint reading a newly added credential through `!Env` on a running worker gets an empty string and creates a provider that answers 404 on its discovery endpoint. Bumping `secret-generation` restarts the worker in the sync that adds the credential; `kubectl rollout restart` fixes a running cluster but leaves nothing for the next one.

> [!CAUTION]
>
> **A workload blueprint is code with admin rights**
>
> The worker applies a blueprint with full access to Authentik's models, so a blueprint ConfigMap from the workloads repository can create a superuser or add someone to `argocd-admins`. The admission policy limits which ConfigMap a workload may write, not what it says: review a blueprint change in the workloads repository like a change here.

> [!WARNING]
>
> **OAuth2 providers must name their grant types**
>
> `grant_types` defaults to an empty list. A provider that omits it serves no grants, and every login fails with `invalid_request: The request is otherwise malformed`.

> [!WARNING]
>
> **The outpost entry replaces its provider list**
>
> `authentik_outposts.outpost` sets `providers` wholesale. Every proxied application must be listed there; adding one and forgetting the list silently unassigns the others, which fail open.

> [!NOTE]
>
> **Home Assistant and Umami are the exceptions**
>
> Each has a login of its own and upstream ships no OIDC provider to replace it, so the outpost sits *in front of* it: browser users authenticate twice, which is defence in depth rather than single sign-on. Home Assistant's companion apps and webhooks hold a long-lived token and cannot complete an interactive login, so `skip_path_regex` lets `/api/`, `/auth/token` and the external-auth callback through; Home Assistant's own accounts are the only thing guarding those paths, and revoking an Authentik account does not revoke a Home Assistant token. Umami's tracker script and collect endpoint are loaded by every visitor of every site that embeds it, so `/script.js` and `/api/send` pass the same way; the rest of the dashboard and API stays behind the login.

## Recovery

### When Authentik is down

Single sign-on is a single point of failure for logging in at all. The ArgoCD path needs a working `kubectl`, so keep a kubeconfig off-cluster.

**ArgoCD** — re-enable the local admin account by merging a pull request that sets `admin.enabled` in `payload/argocd/values.yaml`; the `argocd` Application self-heals, so a `kubectl patch` on `argocd-cm` is reverted at the next reconcile. ArgoCD syncs the merge without a login. Then restart the server, which generates `argocd-initial-admin-secret`, and read the password as in [Quickstart](<https://homelab.wlkr.ch/quickstart/#setup>). Revert the pull request once Authentik is back.

```bash
kubectl -n argocd rollout restart deploy/argocd-server
```

**Grafana** — the login form is hidden, not removed: the admin account works through the API, and `GF_AUTH_DISABLE_LOGIN_FORM=false` brings the form back.

**Proxied dashboards** have no bypass; port-forward instead:

```bash
kubectl -n kube-system port-forward svc/hubble-ui 8080:80
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
```

### ArgoCD's API key account

Disabling the local admin also removes its `apiKey` capability. If a token is needed, add an account scoped to what the token is for rather than re-enabling admin:

The `claude` account is one, for the in-cluster agents; see [Agent account](<https://homelab.wlkr.ch/platform/argocd/#agent-account>). The shape:

```yaml
configs:
  cm:
    accounts.mcp: apiKey
  rbac:
    policy.csv: |
      p, role:mcp, applications, get, */*, allow
      g, mcp, role:mcp
```
