OpenBao¶

OpenBao is the cluster's secret store, an open-source fork of HashiCorp Vault. It holds every secret the cluster consumes, surfaced as native Secret objects by the External Secrets Operator. Sealed, nothing that needs a credential works, and the failure looks like six unrelated things breaking at once.

flowchart LR
    Operator([Operator]) -->|bao CLI| Bao[(OpenBao<br/>KV v2)]
    ESO[External Secrets<br/>Operator] -->|read, write once| Bao
    ESO -->|create/update| KSecret[K8s Secret]
    App[App Pod] -->|env / volume| KSecret

    style Bao fill:#e1f5ff,stroke:#0288d1

At a glance¶

Configuration¶

Audit devices¶

Two, declared as audit stanzas in the server config in application.yaml (this OpenBao refuses bao audit enable over the API): a file device on the audit PVC, and a second file device on stdout, which Alloy ships to Loki with the rest of the container's output. Without one, who read which secret is recorded nowhere; with only one, a full PVC would make OpenBao refuse every request, because a request no enabled device can log is refused. Values are HMACed in both; paths and identities are not. The config is read at start-up, and the StatefulSet updates OnDelete, so a change lands when each pod next restarts — see Unsealing after a restart.

OpenBaoSecretReadOutsideEso in loki-rules.yaml fires on a kv/data/ read by anything but the External Secrets Operator's roles: the operator, a generated root token, or a token that should not exist. A standby audits a request before it resolves the token, so its please forward to the active node rejections carry no identity; the rule drops them, and every ClusterSecretStore targets the openbao-active Service so ESO never produces one.

{namespace="openbao", container="openbao"} |= `"type":"response"` |= `kv/data/` | json | __error__=""

KV layout¶

One KV v2 engine at kv/; every leaf is <workload>/<purpose>, where a workload's prefix is its namespace, the only one its store reaches. ExternalSecrets reference cert-manager/route53 without the data/ prefix ESO adds itself. A path is written by exactly one of two things: a PushSecret beside the ExternalSecret that reads it, when every value is random — see Generated secrets — or make bao-secrets (an agent session's PAT: make claude-session-pat), when a value belongs to an account outside the cluster. A path never mixes the two, because a bao kv put replaces the whole path.

The two route53 leaves are separate IAM users on purpose: cert-manager only ever writes _acme-challenge TXT records, though its policy still allows any record change, while external-dns creates and deletes A records by design, so stolen it can repoint hostnames. One shared key collapses both into "someone owns your domain".

Generated secrets¶

Every random value — session keys, break-glass admin passwords, OIDC client pairs — comes from an ESO Password generator, written into OpenBao by a PushSecret in the same file as the ExternalSecret that reads it back: payload/platform/authentik/secrets.yaml here, <app>/secrets.yaml in the workloads repository. A fresh cluster needs no prompt for them, and a workload that adds a path needs no change to this repository.

ESO refuses to write into a path it did not create (secret not managed by external-secrets), which keeps it away from everything make bao-secrets wrote. Reading a generated value, such as the akadmin password:

kubectl -n openbao exec openbao-0 -- bao kv get -mount=kv -field=bootstrap-password authentik/config

Operator credential¶

No root token outlives the command that needs it. make bao-init ends by creating a userpass login for the scripts and revoking the root token; the password lands in output/credentials/openbao-operator.json (mode 0600). The policy and login are written by bao_create_operator in scripts/lib/bao.sh.

Kubernetes auth¶

ESO authenticates with a short-lived ServiceAccount JWT, which OpenBao validates against the TokenReview API (rbac.yaml binds system:auth-delegator to the openbao ServiceAccount) and answers with a token bound to one policy. Three ClusterSecretStores in cluster-secret-store.yaml each log in with their own role, so what a namespace can reach follows from the store it is allowed to use:

The ServiceAccount the platform store uses lives in cluster-secret-store.yaml too, because the stores only validate against a running, unsealed OpenBao. make bao-init writes the policies and roles on a fresh cluster by running scripts/bao-policies.sh; after a change to that script, make bao-policies rewrites them on a live one. It needs a root token, because the operator login cannot write policies: run it before make bao-rotate-root, or with a generated root token.

Usage¶

Bootstrap¶

1. Initialise, unseal and configure the engine and ESO's auth. Safe to re-run: each configuration step is skipped if already in place, and an initialised cluster is left alone.

   make bao-init
   

   Move the unseal keys before you do anything else

   make bao-init writes the unseal keys to output/credentials/openbao-init.json and the operator login to openbao-operator.json beside it (mode 0600, gitignored) — a plaintext copy of the keys to every secret the cluster holds, next to the etcd encryption key. The root token in the first file is revoked before the command ends. Copy both into a password manager that does not need this cluster to be running, then delete the key file. Losing the keys means the data is unrecoverable: no support line, no recovery flow.

2. Write the values that belong to outside accounts — the prompts follow Quickstart step 11. The generated paths need nothing: their PushSecrets write them once the store validates.

   make bao-secrets
   

3. Confirm the stores validate:

   kubectl get clustersecretstore
   

Authenticating locally¶

kubectl -n openbao port-forward svc/openbao-active 8200:8200 &   # a standby rejects requests
export BAO_ADDR=http://127.0.0.1:8200
bao login -method=userpass username=operator   # password from openbao-operator.json

Storing and reading a secret¶

bao kv put kv/cert-manager/route53 access-key-id='AKIA...' secret-access-key='...'
printf '%s' '{"access-key-id":"AKIA...","secret-access-key":"..."}' \
  | bao kv put -mount=kv cert-manager/route53 -     # what make bao-secrets does: JSON on stdin
bao kv get kv/cert-manager/route53

Rotating a credential¶

A typed value is rotated with make bao-secrets, which asks per existing path whether to overwrite it; a generated one by deleting its path, after which the PushSecret writes a whole new set within its refreshInterval.

1. Rotate the path. For a typed value, answer y for that path; the others are left alone. FORCE=1 make bao-secrets overwrites every ordinary path without asking; kneadtime/config still wants a typed OVERWRITE, and without a terminal every existing path is left alone.

   make bao-secrets
   

   For a generated path, delete it. Every key in it changes, with the consequences the reading component's page lists — for authentik/config that is every session, token and platform OIDC client at once:

   kubectl -n openbao exec openbao-0 -- bao kv metadata delete -mount=kv <path>
   

2. Push the new value into the Secret now rather than at the next refreshInterval — see External Secrets:

   kubectl annotate externalsecret -n <namespace> <name> force-sync=$(date +%s) --overwrite
   

3. Restart whatever read the old value into an environment variable; a rotated Secret reaches a running pod only through a rollout.

Health check¶

for pod in $(kubectl -n openbao get pods -l app.kubernetes.io/name=openbao -o name); do
  kubectl -n openbao exec "$pod" -- bao status
done
kubectl get clustersecretstore
kubectl get externalsecret -A

bao status should show Initialized: true, Sealed: false, and HA Mode: active on one pod with standby on the others; it exits 0 unsealed, 2 sealed and 1 unreachable. Every ClusterSecretStore should report Ready=True.

prometheusrule.yaml mails OpenBaoSealed for each pod sealed past the rule's for, read from the openbao-sealed pod label because a sealed pod still reports Ready, and OpenBaoNoActiveNode when no unsealed active pod is scraped at all.

Pitfalls¶

Recovery¶

Unsealing after a restart¶

make bao-unseal

It feeds a threshold of shares from output/credentials/openbao-init.json to each sealed replica, waits for one that has not joined Raft yet, and is idempotent. With the key file deleted, which is the correct end state, unseal by hand:

for pod in $(kubectl -n openbao get pods -l app.kubernetes.io/name=openbao -o name); do
  kubectl -n openbao exec "$pod" -- bao status >/dev/null 2>&1
  case $? in
    0) echo "$pod: already unsealed" ;;
    2) for i in 1 2 3; do
         kubectl -n openbao exec -it "$pod" -- bao operator unseal
       done ;;
    *) echo "$pod: OpenBao did not answer, check the pod" ;;
  esac
done

A replica that says Vault is not initialized has not joined the Raft cluster — one that predates the retry_join stanzas or started before openbao-0 was initialised. Point it at the leader once; it takes unseal keys afterwards:

kubectl -n openbao exec openbao-1 -- \
  bao operator raft join http://openbao-0.openbao-internal:8200

Generating a root token¶

For what the operator policy does not cover — a policy, an auth method, a snapshot restore, or the operator login itself broken. Needs the operator login and a threshold of unseal shares; each step runs in the pod.

1. Log in as the operator; it prompts for the password from openbao-operator.json.

   kubectl -n openbao exec -it openbao-0 -- bao login -method=userpass username=operator
   

2. Generate a one-time password and start the attempt with it; note both the OTP and the nonce.

   kubectl -n openbao exec openbao-0 -- bao operator generate-root -generate-otp
   kubectl -n openbao exec openbao-0 -- bao operator generate-root -init -otp=<otp>
   

3. Enter one share per run until the threshold is met; the last prints Encoded Token.

   kubectl -n openbao exec -it openbao-0 -- bao operator generate-root -nonce=<nonce>
   

4. Decode it into the root token.

   kubectl -n openbao exec openbao-0 -- bao operator generate-root -decode=<encoded-token> -otp=<otp>
   

5. Do the one job with it, then revoke it and clear the pod's token cache.

   kubectl -n openbao exec -i openbao-0 -- bao login -     # paste the root token, then Ctrl-D
   kubectl -n openbao exec openbao-0 -- bao token revoke -self
   kubectl -n openbao exec openbao-0 -- sh -c 'rm -f "$HOME/.bao-token"'
   

With the operator login gone too, no token can start an attempt: set disable_unauthed_generate_root_endpoints = false in the listener stanza in application.yaml, restart and unseal the pods, and drive the deprecated sys/generate-root/* API with the shares; revert the setting afterwards.

To recreate the operator login with that root token, put it as root_token into a copy of openbao-init.json, move openbao-operator.json aside, and run make bao-rotate-root, which writes a new password and revokes the token.

Backups¶

Logged in as the operator through the openbao-active port-forward from Authenticating locally; its policy covers the snapshot, and a standby refuses it:

bao operator raft snapshot save snapshot.bao

The snapshot holds all KV data and OpenBao's own configuration; restore with bao operator raft snapshot restore. Store it off-cluster — it is encrypted with a key that exists only in the unseal shares, so it is exactly as recoverable as your key custody. See Backups & Recovery.