Repository navigation
backups: postgres round-trip fails on a plugin/barmanObjectStore conflict, and the error comes from code deleted in July #4262
Description
Describe the bug
postgres-2-backup-roundtrip fails because something writes the legacy spec.backup.barmanObjectStore onto a Cluster the chart has already configured through the barman-cloud plugin, and CNPG's webhook refuses to hold both.
The part worth more than the failure itself: the error text comes from code that was deleted from this repository on 2026-07-15, and none of the images that ran in that lane should contain it.
Environment
- Cozystack:
main, in-tree e2e lane - Seen on run 34909258747, suite
postgres-2-backup-roundtrip
To Reproduce
Run the e2e lane and let it reach the postgres backup round-trip.
Expected behaviour
The backup path configures the Cluster one way. Today the chart chooses the plugin, so nothing should be applying barmanObjectStore beside it.
Actual behaviour
From kubectl describe backupjobs -n tenant-root, on BackupJob/pg-src-adhoc:
Message: failed to apply barmanObjectStore to Cluster: admission webhook "vcluster.cnpg.io"
denied the request: Cluster.cluster.cnpg.io "postgres-pg-src" is invalid: spec.plugins:
Invalid value: ["barman-cloud.cloudnative-pg.io"]: Cannot enable a WAL archiver plugin
when barmanObjectStore is configured
The chart's half is the plugin, packages/apps/postgres/templates/db.yaml:173-177:
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: {{ .Release.Name }}And the tree's CNPG driver no longer writes the legacy field at all; it renders an ObjectStore resource instead (internal/backupcontroller/cnpgstrategy_controller.go:484):
objStore.Spec = cnpgtypes.ObjectStoreSpec{
Configuration: *buildBarmanObjectStore(t.BarmanObjectStore, ""),Additional context
Where the message cannot have come from. failed to apply barmanObjectStore to Cluster was removed by 13ff34027 ("fix(backups): migrate CNPG backups to the barman-cloud plugin", 2026-07-15):
$ git show 13ff34027 | grep -E '^[-+].*barmanObjectStore to'
- return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to apply barmanObjectStore to Cluster: %v", err))
It is absent from every revision I can reach:
$ git grep -c "failed to apply barmanObjectStore" origin/main # nothing
$ git grep -c "failed to apply barmanObjectStore" origin/release-1.6 # nothing
$ git grep -c "failed to apply barmanObjectStore" v1.6.0 # nothing
And the two controllers that handle a BackupJob in that run were:
backup-controller-597b56cd6b-pws4t iad.ocir.io/.../backup-controller:main
backupstrategy-controller-5cf66566b-ckcf ghcr.io/.../backupstrategy-controller:v1.6.0
The v1.6.0 image was built 2026-07-21, six days after the migration, so it should carry it too.
So one of three things is true, and I cannot tell which without registry access:
- the image tagged
v1.6.0was not built from the commit that tag names; - a third component emits this and I have not found it;
- the string survives somewhere my grep does not reach.
Any of them is answerable in a minute by someone who can pull the image and grep the binary, which is the concrete ask here.
Why this is worth more than one red suite: a lane running code that exists in no reachable revision undermines every result it reports, not just this one. It also sits next to #4257, where six packages keep release images because their values drifted from the artifact — same family of question, which image is actually running.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
scooby87 commentedon Sep 15, 2026
Investigated the in-repo angle. Confirmed the current tree cannot produce this failure, so the
barmanObjectStore-beside-plugin conflict can only come from a pre-migration binary running in the lane — the same root cause settled in #4258 (the publishedv1.6.0image was built from pre-migration source).The string is gone from every reachable ref.
failed to apply barmanObjectStore to Clusterwas removed in13ff34027(2026-07-16, an ancestor ofv1.6.0) and is absent fromorigin/main,origin/release-1.6andv1.6.0; on all three the call site now readsfailed to attach barman-cloud plugin to Cluster. A grep across the whole tree (Go, charts,hack/*.sh, e2e fixtures) finds no emitter of the old string.The current code cannot write
spec.backup.barmanObjectStorebeside the plugin.applyClusterPluginBackup(internal/backupcontroller/cnpgstrategy_controller.go) SSA-patches onlyCluster.spec.pluginsand separately applies anObjectStoreCR; its field manager owns just those two. Thepatched.Spec.Backup.*writes elsewhere in the file target the tenant Postgres app CR (the recovery path), not the CNPGCluster. The chart (packages/apps/postgres/templates/db.yaml:173-177) renders only the plugin under its barman branch. So the webhook conflict "plugin + barmanObjectStore" is physically unreachable from this code — it can only come from a pre-13ff34027binary.Controller topology names the culprit.
BackupJobReconciler(which emits this message) is registered only incmd/backupstrategy-controller;cmd/backup-controller(the:mainimage) runs only thePlanReconcilerand never touches aBackupJob. So the failing reconcile came from the pinnedbackupstrategy-controller:v1.6.0@sha256:e8bf0521…, which #4258 proves is a pre-migration build.Conclusion: reading #1 from the issue is confirmed — the image tagged
v1.6.0was not built from the commit the tag names. No in-repo change fixes this; the remedy is a registry-side rebuild/re-stamp ofv1.6.0(see #4258 for the binary-level proof and a suggested provenance gate). Related to #4257, which covers a different facet of the same frozen pin.Minor, unrelated: the comment at
internal/backupcontroller/cnpgstrategy_controller_test.go:57still references the old method nameapplyClusterBarmanObjectStore(renamed toapplyClusterPluginBackup) — cosmetic, not a bug.lexfrei commentedon Sep 23, 2026
Duplicate of #4258. The message comes from the
backupstrategy-controller:v1.6.0image, which was built before the migration. Since #4261 the e2e lane runs that controller from the current main image, and the rebuild of the published tag is tracked in #4258.