CloudNativePG¶

Every chart that needs PostgreSQL ships its own, each upgraded by whoever maintains the chart that buried it. CloudNativePG is one operator in cnpg-system and a Cluster resource per database.

At a glance¶

Configuration¶

A Cluster is not a StatefulSet with a nicer name:

Major-version upgrades are declarative too, but offline: ArgoCD reports Degraded while one runs and the workload is down for its duration. Read the upstream guide before merging one.

Databases claim rook-ceph-block, so Ceph's three copies already sit under every database and a single-instance Cluster is a defensible default: the data survives a node loss even when the process does not. Where a workload cannot tolerate the restart, instances: 3 is the change, as Authentik shows.

The operator itself runs two replicas (replicaCount in application.yaml's valuesObject), spread per Replica placement: the same pod serves the Cluster and ScheduledBackup admission webhooks with failurePolicy: Fail, so a single replica rescheduling through a Kured drain refuses every apply, while leader election keeps one of the two reconciling. The chart ships no PodDisruptionBudget.

Backups¶

The Barman Cloud plugin runs beside the operator and injects a sidecar into every instance that names an ObjectStore. The sidecar archives each WAL segment as PostgreSQL closes it and takes the base backups a ScheduledBackup asks for, so a database restores to any second inside the retention window, not just to last night.

ObjectStore is a namespaced resource with no cluster-scoped counterpart, so the per-database part is a short, identical stanza (below); everything else is shared: the plugin, and the cnpg-backups policy in payload/platform/security/network-policies/cluster-wide.yaml that lets any CloudNativePG pod reach the object store.

Usage¶

The contract¶

Every workload that needs PostgreSQL gets a CloudNativePG Cluster. No chart-bundled database, ever. A bundled Postgres moves when the application's chart decides — a major version jump inside someone else's patch release; a Cluster moves when its imageName changes, a Renovate PR with a human in front of it. Most bundled subcharts are also Bitnami's, whose registry terms keep changing. The rule for the workloads repository:

1. Disable the subchart (postgresql.enabled: false or internalDatabase.enabled: false) and point the chart at the Cluster.
2. Never copy the password. CloudNativePG writes username, password, host, port, dbname and a ready-made uri to <cluster-name>-app; charts take those through an existingSecret block, and applications without one read uri as an environment variable. Nothing goes into Git or OpenBao — nothing outside the cluster ever needs this secret.
3. Put the Cluster in the workload's namespace, not cnpg-system, so the namespace's CiliumNetworkPolicy governs port 5432 and deleting the namespace takes the database with it.
4. Set enablePDB: false. The operator's <cluster>-primary PodDisruptionBudget refuses the primary's eviction outright, so a node drain hangs on it until Kured gives up and the node is never rebooted. Without the budget a single instance is down for the move to another node and a three-instance cluster fails over within a minute.
5. Back it up with the stanza below. Velero's nightly PVC snapshot is crash-consistent and restores only to the night it was taken; the WAL archive restores to any second.

Backing up a database¶

Next to the Cluster, in the same namespace, with <db> the cluster's name. The bucket name is cluster-wide, hence the namespace in it; the sync wave puts the store ahead of the Cluster that names it.

---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: <db>-backup
  namespace: <namespace>
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
    argocd.argoproj.io/sync-wave: "-2"
spec:
  bucketName: cnpg-<namespace>
  storageClassName: ceph-bucket
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: <db>
  namespace: <namespace>
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: "-2"
spec:
  retentionPolicy: 14d
  configuration:
    destinationPath: s3://cnpg-<namespace>/
    endpointURL: https://s3.infra.k8s.wlkr.ch
    s3Credentials:
      accessKeyId: {name: <db>-backup, key: AWS_ACCESS_KEY_ID}
      secretAccessKey: {name: <db>-backup, key: AWS_SECRET_ACCESS_KEY}
    wal:
      compression: gzip
    data:
      compression: gzip
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: <db>
  namespace: <namespace>
spec:
  schedule: "0 0 3 * * *"
  immediate: true
  backupOwnerReference: self
  cluster:
    name: <db>
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

And in the Cluster:

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      enabled: true
      isWALArchiver: true
      parameters:
        barmanObjectName: <db>

enabled: true is the webhook's default; written out, ArgoCD's diff matches the live object, since plugins is compared as a whole list. Adding the plugin restarts every instance to inject the sidecar: a single-instance database is down for that restart, a replicated one switches over. immediate: true takes the first base backup at once, so the WAL archive is restorable from the first day.

Health check¶

kubectl get clusters.postgresql.cnpg.io -A
kubectl cnpg status -n <namespace> <cluster-name>

Cluster in healthy state is the phrase; ArgoCD's built-in health check reads the same field, which lets a sync wave wait for a database before the application in front of it. The plugin prints the primary, replication lag, WAL position and last failover.

kubectl get backups.postgresql.cnpg.io -A           # Phase completed, one per night
kubectl get clusters.postgresql.cnpg.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?(@.type=="ContinuousArchiving")].status'
kubectl get objectstores.barmancloud.cnpg.io -A -o yaml | grep -A3 serverRecoveryWindow

ContinuousArchiving True means the last WAL segment reached the bucket; False means WAL is piling up on the instance's volume, which fills it. The recovery window lives on the ObjectStore, per server name: the Cluster's own backup fields and the cnpg_collector_*_backup_timestamp metrics stay empty with a plugin, which upstream documents as deprecated under CNPG-I. prometheusrule.yaml therefore reads the ObjectStore and Backup objects through kube-state-metrics' custom resource metrics: CNPGBackupStale when a server's last successful base backup is older than the rule's threshold, CNPGBackupFailed when a recent Backup ended in failed. Both are warnings on the daily mail.

Pitfalls¶

• A stuck archive fills the volume. PostgreSQL keeps every WAL segment the sidecar has not archived; a broken bucket or policy shows as ContinuousArchiving False long before the PVC is full. Fix the cause, never delete pg_wal.
• A dead replica fills the primary. Its replication slot holds WAL until the replica catches up, so every multi-instance Cluster caps it with max_slot_wal_keep_size at a quarter of the volume; past that the slot is invalidated and the replica restores from the WAL archive or must be recreated with kubectl cnpg destroy <cluster> <instance>.
• A full volume is not resized by the operator. An instance stopped with no free disk space for WALs keeps its PVC size although storage.size grew; patch that PVC to the new size by hand, as upstream documents under "Disk full failure".
• A restore is a new Cluster. Recovery bootstraps a fresh cluster from the bucket; it never rewinds a running one. Give the new cluster its own ObjectStore destination, or its archive would mix with the one it was restored from.

Recovery¶

Point-in-time restore¶

1. Pick the target time (UTC) and check it lies inside the window:

kubectl -n <namespace> get objectstore <db> -o jsonpath='{.status.serverRecoveryWindow}{"\n"}'

1. In the workloads repository, add a second Cluster named <db>-restore beside the original, which stays untouched: every Application prunes, so a renamed Cluster is a deleted one. It bootstraps from the old store, keeping the old name as serverName:

  bootstrap:
    recovery:
      source: origin
      recoveryTarget:
        targetTime: "2026-10-21 12:00:00+00"
  externalClusters:
    - name: origin
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: <db>
          serverName: <db>

Its own plugins entry keeps the same barmanObjectName with serverName: <db>-restore, so the new archive lands beside the old one.

1. Merge and wait for Cluster in healthy state; both databases now run, and the application still reads the original.

2. In a second PR, point the application at <db>-restore-app, set the ScheduledBackup's cluster.name to <db>-restore, and delete the original Cluster; the sync prunes it.