bug(backups/postgres): repeat in-place RestoreJob silently succeeds without restoring (destructive purge skipped when target still carries bootstrap.recovery) #3311

Summary

A repeat in-place PostgreSQL RestoreJob into a target that was already
restored once silently no-ops: the job reports Succeeded and sets the
TargetPurged condition message "Cluster ... deleted; awaiting chart-rendered
re-bootstrap", but the destructive purge never runs. The live cnpg.io Cluster
and its PVCs are never deleted, no *-full-recovery pods appear, and the
database keeps its pre-restore contents.

Verified against main (internal/backupcontroller/cnpgstrategy_controller.go).

Reproduction

1. Back up a postgres app (e.g. 3000 rows).
2. Restore it once in-place. Works: data restored.
3. Dirty the DB (e.g. add rows, now 3100 with garbage).
4. Run a second in-place RestoreJob from the same backup.

Expected: DB purged and re-bootstrapped from the backup, back to 3000 rows.

Actual: RestoreJob = Succeeded, condition message "Cluster deleted; awaiting
re-bootstrap", but:

• PVC unchanged (same uid), disk not recreated;
• no *-full-recovery pods;
• DB still has the 3100 dirty rows.

Root cause

The restore patch sets spec.bootstrap.enabled=true on the Postgres app and it
is never reset:

• buildPostgresAppRestorePatch sets patched.Spec.Bootstrap.Enabled = true
  (cnpgstrategy_controller.go:861); no code path clears it.
• While that flag is true, the chart keeps rendering spec.bootstrap.recovery
  on the live cnpg.io Cluster (packages/apps/postgres/templates/db.yaml:162,
  {{- if .Values.bootstrap.enabled }}).

On the second RestoreJob the re-purge guard misreads this leftover as "an
earlier reconcile of this restore already purged":

• clusterHasRecoveryBootstrap returns true because
  cluster.Spec.Bootstrap.Recovery != nil (:1157), left over from restore Remove would violate PodSecurity "restricted:latest" warnings #1.
• cnpgPurgeNeeded(purgedCondition=false, liveClusterHasRecovery=true) returns
  !true == false, so purgeExistingCluster (Cluster + PVC delete) is skipped
  (:322-327, guard at :622).
• The reconcile still sets TargetPurged=True with the misleading "Cluster
  deleted; awaiting chart-rendered re-bootstrap" message (:729-734); on the
  next pass the still-healthy old cluster passes cnpgClusterHealthy and the
  job is marked Succeeded (:749-773).

The liveClusterHasRecovery guard was added to avoid re-purging a
freshly-recovered cluster when the post-purge Status().Update races/fails. It
cannot distinguish that race from "a previous, completed restore left
bootstrap.recovery on the cluster." The wrong assumption is locked in by a
passing test: TestCNPGPurgeNeeded case "live cluster already recovered (status
write raced): skip - the bug fix" (cnpgstrategy_controller_test.go:573-578)
asserts skip for exactly purgedCondition=false, hasRecovery=true.

Impact

Silent data-integrity failure: a restore reports success while doing nothing. A
tenant using restore to roll back bad data gets a false Succeeded and keeps
the corrupted data.

Proposed fix (direction)

Distinguish "recovery bootstrap left by THIS RestoreJob's own purge" from
"recovery bootstrap left by a previous completed restore":

1. Record the purged Cluster's UID/creationTimestamp on the RestoreJob status at
   purge time; skip re-purge only when the live Cluster is a different
   (chart-recreated) object. (Safer.)
2. Reset spec.bootstrap.enabled=false after a successful restore - must reuse
   the same HR-suspend flow, since flipping bootstrap on an initialized Cluster
   hits CNPG's "Only one bootstrap" admission.

Either way: add an integration/e2e test for the repeat-in-place-restore
scenario and fix the TestCNPGPurgeNeeded case that encodes the wrong
assumption.

Related

Related to #2774 (PITR tracking epic). That issue lists the mirror failure mode
(recovery killed mid-flight by a Flux re-trigger); this is the opposite - the
purge is skipped so recovery never runs.