fix(postgres): in-place restore loses newServerName to a stale Cluster read #4298

What happened

An in-place RestoreJob against a Postgres application fails, and the RestoreJob ends on its own deadline:

RestoreJob exceeded 30m0s deadline before target Cluster reached a healthy state

The full-recovery job underneath never starts, because barman-cloud-check-wal-archive refuses the destination:

WAL archive check failed for server postgres-postgres: Expected empty archive

That is the failure bootstrap.newServerName was introduced to prevent in v1.6.3 (#3988). The mechanism itself works — the Postgres CR carries a fresh newServerName, and the chart renders the Cluster with it — but the value does not survive on the Cluster.

Why

applyClusterPluginBackup reads the Cluster through the manager's cached client and then preserves the serverName it finds there, which is correct in intent: flipping the prefix mid-stream would split the archive.

The read is not reliable in this window, though. BackupJobReconciler.SetupWithManager registers For(&backupsv1alpha1.BackupJob{}) and nothing else, so a Cluster only reaches the cache through a lazily started informer with no resync tied to the Cluster's own changes. An in-place restore deletes the Cluster and the chart re-renders it seconds later, so a read served during that window describes the object that was purged — and applyClusterPluginBackup writes its serverName back over the freshly rendered one.

The same stale read carries a stale UID, which the function uses as the owner reference of the ObjectStore it applies. An ObjectStore owned by a Cluster that no longer exists is garbage-collected as soon as it is written.

Observed

Field managers on the Cluster during a failing restore, twice on separate runs with the same 4-second gap:

11:31:00  restore starts
11:31:03  helm-controller                Apply   <- Cluster rendered with newServerName
11:31:07  cozystack-cnpg-backup-driver   Apply   <- serverName reverted to the source
11:31:12  manager                        Update

The divergence is visible in two commands:

kubectl -n <ns> get postgres <app> -o jsonpath='{.spec.bootstrap.newServerName}'
# postgres-postgres-restore-7235461b0c50fa3a

kubectl -n <ns> get cluster.postgresql.cnpg.io <cluster> -o jsonpath='{.spec.plugins[0].parameters.serverName}'
# postgres-postgres

Rewriting the Cluster's serverName to the CR's newServerName while the restore is still running lets the same restore finish normally.

Version

v1.6.3, and main carries the same code.

Workaround

None that suits an automated flow. The manual rewrite above works but has to be timed inside the restore.