Summary
CNPG backups of system PostgreSQL databases are non-functional on 1.6: the backup config uses the native spec.backup.barmanObjectStore, but the pinned PostgreSQL image variant no longer ships the barman-cloud-* binaries, so backups fail with:
LastBackupFailed: exec: "barman-cloud-backup": executable file not found in $PATH
WAL archiving is affected too (no barman-cloud-wal-archive either) — beyond failing base backups, a failing archive_command blocks WAL recycling and risks filling the instance disk.
Versions
- cozystack
v1.6.0-rc.x
- CNPG operator
1.27.1 (chart cloudnative-pg 0.26.1; observed 1.27.3 on cluster)
- PostgreSQL image
ghcr.io/cloudnative-pg/postgresql:17.7-standard-trixie (PostgreSQL 17.7, standard variant)
Root cause
#2342 moved system Postgres to the CNPG standard image variant (via ClusterImageCatalog). Per that PR's own variant table:
- standard: minimal + pgaudit + extensions ✅ "We use this"
- system: standard + Barman Cloud binaries (deprecated) ❌
So the chosen standard variant intentionally excludes the barman-cloud-* tools (native Barman Cloud is deprecated in CNPG 1.27, removed in 1.29 — the replacement is the Barman Cloud Plugin). But cozystack still configures native spec.backup.barmanObjectStore, which requires those binaries in the image, in at least:
Migration 37 pins all system DBs (keycloak-db, grafana-db, alerta-db, seaweedfs-db, harbor DB) to 17.7-standard-trixie, so the whole system-Postgres backup surface is affected; the cozy-default-cnpg strategy extends the breakage to tenant Postgres apps that run on the barman-less image.
Reproduction
On a 1.6 cluster, enable a barman backup on any system PG DB (e.g. keycloak backup.enabled=true + destinationPath/creds), then trigger a Backup:
$ kubectl -n cozy-keycloak get backup keycloak-db-test -o jsonpath='{.status.error}'
instance manager was restarted during backup on pod keycloak-db-2
$ kubectl -n cozy-keycloak get cluster keycloak-db -o jsonpath='{.status.conditions[?(@.type=="LastBackupSucceeded")].message}'
exec: "barman-cloud-backup": executable file not found in $PATH
Confirmed the image has no barman tools:
$ kubectl -n cozy-keycloak exec keycloak-db-1 -c postgres -- sh -c 'ls /usr/bin/barman* /usr/local/bin/barman* 2>&1; which barman-cloud-backup'
# (nothing)
Suggested fix
Adopt the CNPG Barman Cloud Plugin across the CNPG-backup surface: install the plugin (operator + ObjectStore CRD) and convert spec.backup.barmanObjectStore → spec.plugins: [{name: barman-cloud.cloudnative-pg.io, ...}] + an ObjectStore CR in system/keycloak, system/backupstrategy-controller (cozy-default-cnpg), and apps/postgres. (Alternatively ship an image variant that still bundles barman-cloud, but that runs counter to CloudNativePG's direction.)
Related: #2342 (image variant change, the cause), #3174 (keycloak backup, affected).
Found while testing #3174 keycloak backups on a freedom-portal cluster (cozystack 1.6).
Summary
CNPG backups of system PostgreSQL databases are non-functional on 1.6: the backup config uses the native
spec.backup.barmanObjectStore, but the pinned PostgreSQL image variant no longer ships thebarman-cloud-*binaries, so backups fail with:WAL archiving is affected too (no
barman-cloud-wal-archiveeither) — beyond failing base backups, a failingarchive_commandblocks WAL recycling and risks filling the instance disk.Versions
v1.6.0-rc.x1.27.1(chartcloudnative-pg0.26.1; observed 1.27.3 on cluster)ghcr.io/cloudnative-pg/postgresql:17.7-standard-trixie(PostgreSQL 17.7, standard variant)Root cause
#2342 moved system Postgres to the CNPG standard image variant (via
ClusterImageCatalog). Per that PR's own variant table:So the chosen
standardvariant intentionally excludes thebarman-cloud-*tools (native Barman Cloud is deprecated in CNPG 1.27, removed in 1.29 — the replacement is the Barman Cloud Plugin). But cozystack still configures nativespec.backup.barmanObjectStore, which requires those binaries in the image, in at least:packages/system/keycloak(S3 backup, feat(keycloak): add S3 backup for the CNPG database #3174)packages/system/backupstrategy-controller— thecozy-default-cnpgStrategy (strategy-cnpg-default.yaml)packages/apps/postgresMigration
37pins all system DBs (keycloak-db, grafana-db, alerta-db, seaweedfs-db, harbor DB) to17.7-standard-trixie, so the whole system-Postgres backup surface is affected; thecozy-default-cnpgstrategy extends the breakage to tenant Postgres apps that run on the barman-less image.Reproduction
On a 1.6 cluster, enable a barman backup on any system PG DB (e.g. keycloak
backup.enabled=true+ destinationPath/creds), then trigger aBackup:Confirmed the image has no barman tools:
Suggested fix
Adopt the CNPG Barman Cloud Plugin across the CNPG-backup surface: install the plugin (operator +
ObjectStoreCRD) and convertspec.backup.barmanObjectStore→spec.plugins: [{name: barman-cloud.cloudnative-pg.io, ...}]+ anObjectStoreCR insystem/keycloak,system/backupstrategy-controller(cozy-default-cnpg), andapps/postgres. (Alternatively ship an image variant that still bundles barman-cloud, but that runs counter to CloudNativePG's direction.)Related: #2342 (image variant change, the cause), #3174 (keycloak backup, affected).
Found while testing #3174 keycloak backups on a freedom-portal cluster (cozystack 1.6).