OpenBao¶

OpenBao is the cluster's secret store, an open-source fork of HashiCorp Vault. It holds every secret the cluster consumes, surfaced as native Secret objects by the External Secrets Operator. Sealed, nothing that needs a credential works, and the failure looks like six unrelated things breaking at once.

flowchart LR
    Operator([Operator]) -->|bao CLI| Bao[(OpenBao<br/>KV v2)]
    ESO[External Secrets<br/>Operator] -->|read| Bao
    ESO -->|create/update| KSecret[K8s Secret]
    App[App Pod] -->|env / volume| KSecret

    style Bao fill:#e1f5ff,stroke:#0288d1

At a glance¶

Configuration¶

Audit devices¶

Two, declared as audit stanzas in the server config in application.yaml (this OpenBao refuses bao audit enable over the API): a file device on the audit PVC, and a second file device on stdout, which Alloy ships to Loki with the rest of the container's output. Without one, who read which secret is recorded nowhere; with only one, a full PVC would make OpenBao refuse every request, because a request no enabled device can log is refused. Values are HMACed in both; paths and identities are not. The config is read at start-up, and the StatefulSet updates OnDelete, so a change lands when each pod next restarts — see Unsealing after a restart.

OpenBaoSecretReadOutsideEso in loki-rules.yaml fires on a kv/data/ read by anything but the External Secrets Operator's policy: a person with the root token, or a token that should not exist.

{namespace="openbao", container="openbao"} |= `"type":"response"` |= `kv/data/` | json | __error__=""

KV layout¶

One KV v2 engine at kv/; every leaf is <workload>/<purpose>, and ExternalSecrets reference cert-manager/route53 without the data/ prefix ESO adds itself. The bao kv put for each path sits in a comment at the top of the ExternalSecret that consumes it, collected in Quickstart step 11.

The two route53 leaves are separate IAM users on purpose: cert-manager's key only writes _acme-challenge TXT records, so stolen it can issue certificates, while external-dns's creates and deletes A records, so stolen it can repoint hostnames. One shared key collapses both into "someone owns your domain".

Kubernetes auth¶

ESO authenticates with a short-lived ServiceAccount JWT, which OpenBao validates against the TokenReview API (rbac.yaml binds system:auth-delegator to the openbao ServiceAccount) and answers with a token bound to the external-secrets policy. The external-secrets-vault ServiceAccount and the ClusterSecretStore live in cluster-secret-store.yaml here, because the store only validates against a running, unsealed OpenBao. make bao-init configures all of it:

bao secrets enable -path=kv -version=2 kv
bao auth enable kubernetes
bao write auth/kubernetes/config kubernetes_host="https://kubernetes.default.svc"
bao policy write external-secrets - <<'EOF'
path "kv/data/*"     { capabilities = ["read"] }
path "kv/metadata/*" { capabilities = ["read", "list"] }
EOF
bao write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets-vault \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets ttl=1h

Usage¶

Bootstrap¶

1. Initialise, unseal and configure the engine and ESO's auth. Safe to re-run: each configuration step is skipped if already in place, and an initialised cluster is left alone.

   make bao-init
   

   Move the unseal keys before you do anything else

   make bao-init writes the 5 unseal keys and the root token to output/credentials/openbao-init.json (mode 0600, gitignored) — a plaintext copy of the keys to every secret the cluster holds, next to the etcd encryption key. Copy them into a password manager that does not need this cluster to be running, then delete the file. Losing all five means the data is unrecoverable: no support line, no recovery flow.

2. Populate the five paths the cluster reads — the prompts follow Quickstart step 11:

   make bao-secrets
   

3. Confirm the store validates:

   kubectl get clustersecretstore openbao
   

Authenticating locally¶

kubectl -n openbao port-forward svc/openbao 8200:8200 &
export BAO_ADDR=http://127.0.0.1:8200
bao login   # paste the root token

Storing and reading a secret¶

bao kv put kv/cert-manager/route53 access-key-id='AKIA...' secret-access-key='...'
bao kv put -mount=kv cert-manager/route53 @/tmp/secret.json   # what make bao-secrets does
bao kv get kv/cert-manager/route53

Health check¶

kubectl -n openbao exec -it openbao-0 -- bao status
kubectl get clustersecretstore openbao -o yaml
kubectl get externalsecret -A

bao status should show Initialized: true, Sealed: false, and HA Mode: active on one pod with standby on the others; it exits 0 unsealed, 2 sealed and 1 unreachable. The ClusterSecretStore should report Ready=True.

Pitfalls¶

Recovery¶

Unsealing after a restart¶

make bao-unseal

It feeds three shares from output/credentials/openbao-init.json to each sealed replica, waits for one that has not joined Raft yet, and is idempotent. With the key file deleted, which is the correct end state, unseal by hand:

for pod in openbao-0 openbao-1 openbao-2; do
  kubectl -n openbao exec "$pod" -- bao status >/dev/null 2>&1
  case $? in
    0) echo "$pod: already unsealed" ;;
    2) for i in 1 2 3; do
         kubectl -n openbao exec -it "$pod" -- bao operator unseal
       done ;;
    *) echo "$pod: OpenBao did not answer, check the pod" ;;
  esac
done

A replica that says Vault is not initialized has not joined the Raft cluster — one that predates the retry_join stanzas or started before openbao-0 was initialised. Point it at the leader once; it takes unseal keys afterwards:

kubectl -n openbao exec openbao-1 -- \
  bao operator raft join http://openbao-0.openbao-internal:8200

Backups¶

bao operator raft snapshot save snapshot.bao

The snapshot holds all KV data and OpenBao's own configuration; restore with bao operator raft snapshot restore. Store it off-cluster — it is encrypted with a key that exists only in the five shares, so it is exactly as recoverable as your key custody. See Backups & Recovery.