Skip to content

fix(backups): concurrent backups election logic - #10747

Merged
leonardoce merged 3 commits into
cloudnative-pg:mainfrom
GabriFedi97:dev/10622
Jun 3, 2026
Merged

leonardoce merged 3 commits into
cloudnative-pg:mainfrom
GabriFedi97:dev/10622

Conversation

@GabriFedi97

Copy link
Copy Markdown
Member

Summary

  • Fix backup election logic in CanExecuteBackup() that could preempt an
    already-started backup in favor of a newer one whose name sorts earlier
    alphabetically
  • Sort backups by creation time (oldest first) instead of alphabetically, so
    the backup that started first always wins the election
  • Introduce IsExecuting() helper to distinguish backups that have acquired
    resources (started/running) from those still waiting (pending), without
    changing the existing IsInProgress() semantics

The fix replaces alphabetical sorting with creation-time sorting (SortByCreationTimeAndName), ensuring the oldest backup always sorts first. This ensures a newer backup can never reach started phase while an older one is still in progress — the older one would have blocked it via the election check.

Closes #10622

@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.28 labels May 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@GabriFedi97
GabriFedi97 force-pushed the dev/10622 branch 5 times, most recently from 2e1ec4e to 6970055 Compare May 21, 2026 06:50
@GabriFedi97
GabriFedi97 marked this pull request as ready for review May 21, 2026 06:50
@GabriFedi97
GabriFedi97 requested a review from a team May 21, 2026 06:50
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 21, 2026
@armru

armru commented May 27, 2026

Copy link
Copy Markdown
Member

/test

@github-actions

Copy link
Copy Markdown
Contributor

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/26501720222

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 27, 2026
@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label May 27, 2026
GabriFedi97 and others added 3 commits June 3, 2026 14:37
The backup election logic in CanExecuteBackup() sorted backups
alphabetically by name and picked the first "in-progress" one. This
allowed a newer backup with an alphabetically earlier name to preempt a
backup already in started phase. Since started VolumeSnapshot backups
hold a temporary replication slot and a Postgres session on the primary,
preemption leaves these resources orphaned, potentially leading to WAL
disk saturation.

Replace alphabetical sorting with creation-time sorting so the oldest
backup always wins the election. A newer backup can never reach started
phase while an older one is still in progress, making this a complete
fix.

Signed-off-by: Gabriele Fedi <[email protected]>
Add two CanExecuteBackup scenarios that explicitly set
CreationTimestamp, so the regression guard cannot be silently
defeated by the alphabetical tie-breaker:

- an older Started backup must win over a newer Pending backup
  whose name sorts earlier alphabetically (the exact issue cloudnative-pg#10622
  scenario);
- when only Pending backups exist, the oldest one wins regardless
  of name order.

Without explicit timestamps the existing CanExecuteBackup tests
all rely on the name tie-breaker, and a regression replacing
SortByCreationTimeAndName with SortByName inside CanExecuteBackup
would not be detected.

Signed-off-by: Armando Ruocco <[email protected]>
Restate the two acceptance conditions in terms of the new
IsExecuting / oldest-pending logic, and document the name
tie-breaker. The previous wording referenced "first running
backup of the list", which no longer matches the post-fix
behaviour where the executing backup is unique by construction.

Signed-off-by: Armando Ruocco <[email protected]>
@leonardoce
leonardoce merged commit 4fe7715 into cloudnative-pg:main Jun 3, 2026
34 of 35 checks passed
cnpg-bot pushed a commit that referenced this pull request Jun 3, 2026
Previously, when multiple Backup objects existed concurrently, election
among them was decided by alphabetical name order. A newer backup whose
name sorted earlier could preempt one that had already started, leaving
the displaced backup's replication slot and PostgreSQL session orphaned
on the primary.

Backups now run in strict creation-time order, so an already-executing
backup is never preempted by a newer one — regardless of name. Name is
kept only as a deterministic tie-breaker when timestamps match.

Closes #10622

Signed-off-by: Gabriele Fedi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 4fe7715)
leonardoce pushed a commit that referenced this pull request Jun 3, 2026
Previously, when multiple Backup objects existed concurrently, election
among them was decided by alphabetical name order. A newer backup whose
name sorted earlier could preempt one that had already started, leaving
the displaced backup's replication slot and PostgreSQL session orphaned
on the primary.

Backups now run in strict creation-time order, so an already-executing
backup is never preempted by a newer one — regardless of name. Name is
kept only as a deterministic tie-breaker when timestamps match.

Closes #10622

Signed-off-by: Gabriele Fedi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 4fe7715)
leonardoce pushed a commit that referenced this pull request Jun 3, 2026
Previously, when multiple Backup objects existed concurrently, election
among them was decided by alphabetical name order. A newer backup whose
name sorted earlier could preempt one that had already started, leaving
the displaced backup's replication slot and PostgreSQL session orphaned
on the primary.

Backups now run in strict creation-time order, so an already-executing
backup is never preempted by a newer one — regardless of name. Name is
kept only as a deterministic tie-breaker when timestamps match.

Closes #10622

Signed-off-by: Gabriele Fedi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 4fe7715)
(cherry picked from commit 2bd219f)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Jul 7, 2026
##### [\`v1.29.2\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.2)

**Release date:** Jun 29, 2026

##### Important changes

- Updated the deprecation notice for native (in-tree) Barman Cloud support to reflect that it will now be removed in CloudNativePG 1.31.0, rather than 1.30.0. Users are still encouraged to migrate to the Barman Cloud Plugin. ([#11083](cloudnative-pg/cloudnative-pg#11083)) <!-- 1.29 1.28 -->

- The `cluster` reference is now immutable on the `Database`, `Pooler`, `Publication`, `Subscription`, and `ScheduledBackup` resources. Pointing one of these objects at a different cluster has no well-defined semantics and previously left the controllers in an inconsistent state; the update is now rejected at the API server via a CEL validation rule. ([#10743](cloudnative-pg/cloudnative-pg#10743)) <!-- 1.29 1.28 1.25 -->

##### Enhancements

- Enabled `pg_upgrade` in-place major upgrades to PostgreSQL 19 or later for clusters that use Image Volume extensions, building on the extension-path support added to `pg_upgrade` in PostgreSQL 19. During the upgrade `Job`, the source- and target-version extension images are mounted side by side, so the old server keeps its libraries and a failed upgrade reverts cleanly. ([#10366](cloudnative-pg/cloudnative-pg#10366)) <!-- 1.29 -->

- Added a label selector to the `Cluster` scale subresource (`status.selector`), making a `Cluster` a valid `targetRef` for the Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA), which can now map a `Cluster` to its instance pods. Contributed by [@sebv004](https://github.com/sebv004). ([#8996](cloudnative-pg/cloudnative-pg#8996)) <!-- 1.29 1.28 1.25 -->

- The operator now emits a `Warning` `PrimaryStatusCheckFailed` event on the `Cluster` when the primary pod is `Ready` from the kubelet perspective but the operator's `/pg/status` check fails and failover is deferred, giving users visibility into the deferral via `kubectl describe cluster`. ([#10509](cloudnative-pg/cloudnative-pg#10509)) <!-- 1.29 1.28 1.25 -->

- The operator now reloads a CNPG-i plugin automatically when its pods are rolled: it watches the `EndpointSlices` backing plugin `Services` and re-enqueues every cluster using the plugin once the new pods become `Ready`, so an upgraded plugin is picked up without waiting for the next resync. ([#10836](cloudnative-pg/cloudnative-pg#10836)) <!-- 1.29 1.28 1.25 -->

##### Security and Supply Chain

- **`CVE-2026-55769` / `GHSA-x8c2-3p4r-v9r6`: `search_path` pinning on operator-issued connections**: a database owner could plant overloaded built-in operators in the `public` schema and alter the `search_path` so that operator introspection probes, running as the cluster superuser, resolved those overloads before `pg_catalog`, a `CWE-426` privilege-escalation chain (same class as `CVE-2018-1058`) that could lead to in-pod RCE via `COPY ... FROM PROGRAM`. The operator now pins `search_path = pg_catalog, public, pg_temp` on every pooled connection so it ships in the startup message and takes precedence over tenant-controlled defaults. ([#10774](cloudnative-pg/cloudnative-pg#10774), [`GHSA-x8c2-3p4r-v9r6`](GHSA-x8c2-3p4r-v9r6)) <!-- 1.29 1.28 1.25 -->

- **`CVE-2026-55765` / `GHSA-w3gf-xc94-wvmj`: operator-side SCRAM-SHA-256 password encoding**: the operator now SCRAM-SHA-256 encodes cleartext role passwords before issuing `CREATE`/`ALTER ROLE ... PASSWORD`, so the literal PostgreSQL parses (and that extensions such as `pg_stat_statements` or `pgaudit` may capture) is the SCRAM verifier rather than the cleartext secret. Pre-hashed (MD5 or SCRAM) values are forwarded unchanged, and the per-Secret annotation `cnpg.io/passwordPassthrough: "enabled"` opts out. ([#10724](cloudnative-pg/cloudnative-pg#10724), [`GHSA-w3gf-xc94-wvmj`](GHSA-w3gf-xc94-wvmj)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Added support for Kubernetes 1.36. ([#10900](cloudnative-pg/cloudnative-pg#10900)) <!-- 1.29 1.28 -->

- Updated the default PostgreSQL version to 18.4. ([#10719](cloudnative-pg/cloudnative-pg#10719)) <!-- 1.29 1.28 1.25 -->

- Updated the Kubernetes versions used to test the operator on public cloud providers. ([#10720](cloudnative-pg/cloudnative-pg#10720), <!-- 1.29 1.28 1.25 --> [#10563](cloudnative-pg/cloudnative-pg#10563), <!-- 1.29 1.28 1.25 --> [#11033](cloudnative-pg/cloudnative-pg#11033)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed `spec.postgresql.parameters` accepting keys that are not valid PostgreSQL parameter names, which could inject arbitrary directives into `postgresql.conf`; key names are now validated by the webhook. ([#11029](cloudnative-pg/cloudnative-pg#11029)) <!-- 1.29 1.28 1.25 -->

- Fixed declarative `Database`, `Publication`, and `Subscription` objects reporting a stale primary-side status forever after their cluster was demoted to a replica; the controller now re-checks the replica condition and watches the `Cluster` so a demotion is detected promptly. ([#10871](cloudnative-pg/cloudnative-pg#10871)) <!-- 1.29 1.28 1.25 -->

- Fixed non-sequential pod names (for example `-1`, `-3`) caused by the instance serial counter being advanced before the corresponding `Job` and PVCs were created; the bump is now persisted only after those resources exist. ([#10491](cloudnative-pg/cloudnative-pg#10491)) <!-- 1.29 1.28 1.25 -->

- Fixed a switchover deadlock when a WAL-archiver plugin was enabled on an existing cluster: with `primaryUpdateMethod: switchover` the primary could not be rolled out because a clean demotion needs the archiver sidecar that is still missing. The operator now recreates the primary Pod in place so the sidecar is injected and archiving resumes. The check also covers plugins that inject the archiver as a native sidecar (an init container with `restartPolicy: Always`), such as the Barman Cloud plugin. ([#11032](cloudnative-pg/cloudnative-pg#11032), <!-- 1.29 1.28 1.25 --> [#11059](cloudnative-pg/cloudnative-pg#11059)) <!-- 1.29 1.28 1.25 -->

- Fixed a cluster staying in `Setting up primary` indefinitely when the instance-creation Job exhausted its backoff limit; the operator now detects the terminal Job failure and marks the cluster unrecoverable, naming the failed Job and pointing to its logs. ([#11035](cloudnative-pg/cloudnative-pg#11035)) <!-- 1.29 1.28 1.25 -->

- Fixed deletion of a `Database`, `Publication`, or `Subscription` getting stuck in `Terminating` on a replica cluster, where the replica gate ran before the finalizer reconciler and the finalizer was never released. On a replica the PostgreSQL object is left to the primary cluster. ([#10853](cloudnative-pg/cloudnative-pg#10853)) <!-- 1.29 1.28 1.25 -->

- Fixed a conflicting duplicate `Database` or `Subscription` with a `delete` reclaim policy dropping the PostgreSQL object owned by the surviving CR; the drop is now gated on a recorded reconciliation. ([#10870](cloudnative-pg/cloudnative-pg#10870)) <!-- 1.29 1.28 1.25 -->

- Fixed the `postgres` superuser being left locked out after superuser access was disabled and then re-enabled, because the cached secret version was not invalidated and the password was never re-applied. Diagnosed by [@mhartmann-jaconi](https://github.com/mhartmann-jaconi). ([#10834](cloudnative-pg/cloudnative-pg#10834)) <!-- 1.29 1.28 1.25 -->

- Fixed backups getting stuck in the `started` phase when the instance manager running them was restarted (for example by the in-place upgrade following an operator upgrade) before the backup reached `running`; the reconciliation is now rescheduled so the lost session is detected. ([#10859](cloudnative-pg/cloudnative-pg#10859)) <!-- 1.29 1.28 -->

- Fixed `exec`/`attach` streaming to negotiate WebSocket with a SPDY fallback, restoring compatibility both with Kubernetes versions that have removed SPDY and with platforms such as OpenShift that reject WebSocket exec upgrades. Contributed by [@bartscheers](https://github.com/bartscheers). ([#10876](cloudnative-pg/cloudnative-pg#10876), [#10933](cloudnative-pg/cloudnative-pg#10933)) <!-- 1.29 -->

- Fixed resource leaks when concurrent `Backup` objects raced: backups now run in strict creation-time order, so an already-executing backup is never preempted by a newer one and its replication slot and PostgreSQL session are no longer orphaned on the primary. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10747](cloudnative-pg/cloudnative-pg#10747)) <!-- 1.29 1.28 1.25 -->

- Fixed role reconciliation clearing the password on a PostgreSQL role when the referenced Secret could not be fetched; the role is now left untouched until the Secret becomes available, and per-action errors are aggregated for better visibility. ([#10053](cloudnative-pg/cloudnative-pg#10053)) <!-- 1.29 1.28 1.25 -->

- Fixed a bootstrap failure where a metrics-exporter setup error (commonly a duplicate-key race with the controller) rolled back `streaming_replica` creation and wedged replica joins. The metrics-exporter step now runs in a separate transaction. Contributed by [@BlaiseAntony](https://github.com/BlaiseAntony). ([#10749](cloudnative-pg/cloudnative-pg#10749)) <!-- 1.29 1.28 1.25 -->

- Fixed a `ScheduledBackup` controller loop that occurred when a `Backup` was created but its status patch never landed; the controller now adopts an existing `Backup` for the next iteration instead of looping on `AlreadyExists`. ([#10612](cloudnative-pg/cloudnative-pg#10612)) <!-- 1.29 1.28 1.25 -->

- Fixed a nil-pointer panic when reconciling a `Pooler` whose `Cluster` has been deleted. ([#10667](cloudnative-pg/cloudnative-pg#10667)) <!-- 1.29 1.28 -->

- Fixed bootstrap log handling so that all named log pipes (`postgres`, `postgres.csv`, and `postgres.json`) get consumers during `WithActiveInstance`, preventing regular files from being created in place of the named pipes. ([#10043](cloudnative-pg/cloudnative-pg#10043)) <!-- 1.29 1.28 1.25 -->

- Fixed generation of invalid IPv6 URLs by wrapping the address in square brackets. Contributed by [@Infinoid](https://github.com/Infinoid). ([#10682](cloudnative-pg/cloudnative-pg#10682)) <!-- 1.29 1.28 1.25 -->

- Fixed an external cluster plugin still being treated as active when its configuration set `enabled: false`. ([#10932](cloudnative-pg/cloudnative-pg#10932)) <!-- 1.29 1.28 1.25 -->

- Fixed a race during bootstrap recovery from an object store where the restore job could read a stale `Cluster` (primary not yet recorded and timeline still unset) and have its `.history` files rejected by the split-brain guard. When this happened, recovery stopped at the base backup's timeline and silently dropped transactions committed on later timelines. History files are now allowed while the cluster timeline is unset. Contributed by [@dennispidun](https://github.com/dennispidun). ([#10818](cloudnative-pg/cloudnative-pg#10818)) <!-- 1.29 1.28 1.25 -->

- Fixed a cache race during cluster creation when the server and client CA resolve to the same Secret (the default): a stale informer cache triggered a redundant `Create` that failed with `AlreadyExists` and could leave the cluster stuck in `Unable to create required cluster objects`. The operator now reuses the already-fetched CA Secret when the names match. ([#10989](cloudnative-pg/cloudnative-pg#10989)) <!-- 1.29 1.28 1.25 -->

- Fixed the `pg_basebackup` bootstrap path overwriting or failing on a pre-existing `PGDATA` (for example after a replica Pod restart) by enforcing the same pre-flight directory check already applied by the other bootstrap methods; this also protects statically provisioned PVCs from being silently overwritten. ([#11006](cloudnative-pg/cloudnative-pg#11006)) <!-- 1.29 1.28 1.25 -->

- Fixed the `Cluster` phase flapping between `Healthy` and a plugin-failure phase when a post-reconcile plugin hook returned an error; the `Healthy` phase is now registered as the last step of a successful reconciliation, so a loop that ends in a plugin error never reports `Healthy`. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10421](cloudnative-pg/cloudnative-pg#10421)) <!-- 1.29 1.28 1.25 -->

- Fixed stale certificate data and partial reads after an external server's Secret was rotated (for example a CA bundle shrinking from two certificates to one): the file is now written atomically, so libpq always reads either the old or the new value, never a mix. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10975](cloudnative-pg/cloudnative-pg#10975)) <!-- 1.29 1.28 1.25 -->

- Fixed plugin connectivity to use the plugin `Service` FQDN instead of its short name, avoiding failures when a cluster-level proxy is automatically injected into pods. Contributed by [@kdautrey](https://github.com/kdautrey). ([#10921](cloudnative-pg/cloudnative-pg#10921)) <!-- 1.29 1.28 1.25 -->

- Fixed excessive operator log noise from the per-request `Cluster` create/update validation webhook messages, now logged at `debug` instead of `info`. ([#10984](cloudnative-pg/cloudnative-pg#10984)) <!-- 1.29 1.28 1.25 -->

- Fixed a first-primary bootstrap deadlock where a status-patch conflict after the data PVC was created but before the initialization Job was started left the orphan Pending PVC counted as an instance, blocking the bootstrap gate; the PVC-state reconciler now recreates the bootstrap Job reusing the assigned serial. ([#11039](cloudnative-pg/cloudnative-pg#11039)) <!-- 1.29 1.28 1.25 -->

- Fixed external cluster names and secret selector references being joined into filesystem paths without validation, letting a `..` component or path separator escape the external secrets directory when the instance manager dumps connection material; these values are now rejected at the validating webhook and re-checked at the write site. Reported by [@r0binak](https://github.com/r0binak). ([#11045](cloudnative-pg/cloudnative-pg#11045)) <!-- 1.29 1.28 1.25 -->

- Fixed a backup getting stuck in `pending` forever: the concurrent-backup gate ran on every reconcile and could overwrite an already-completed phase written asynchronously by the instance manager. The gate now runs only while the backup phase is still unset or `pending`. ([#11056](cloudnative-pg/cloudnative-pg#11056)) <!-- 1.29 -->

- Fixed a declarative `VolumeSnapshot` backup being permanently marked as failed when a stale cache made the operator re-create a snapshot it had already provisioned, failing with `AlreadyExists`. The operator now tolerates the collision when the existing snapshot carries this backup's label and adopts it; a collision with a foreign snapshot still surfaces as an error. ([#11071](cloudnative-pg/cloudnative-pg#11071)) <!-- 1.29 1.28 1.25 -->

- Fixed a volume snapshot backup being discarded on a transient instance-manager connection error (for example a dial timeout from a brief pod-network disruption) during the finalize step, even when its snapshots were already provisioned; such network errors are now retried instead of treated as terminal. ([#11069](cloudnative-pg/cloudnative-pg#11069)) <!-- 1.29 1.28 1.25 -->

- Fixed a replica switchover losing its `status.demotionToken` when a reconcile was requeued between storing the token and cleaning up the transition metadata (for example a cleanup patch failing against a flaky webhook); the empty no-change token is no longer patched back over the stored value. ([#11075](cloudnative-pg/cloudnative-pg#11075)) <!-- 1.29 1.28 1.25 -->

- `cnpg` plugin:

  - Fixed `kubectl cnpg psql` on Windows, where execution relied on a Unix-only system call and failed with "not supported by windows"; Windows now launches `kubectl exec` as a child process. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#10972](cloudnative-pg/cloudnative-pg#10972)) <!-- 1.29 1.28 1.25 -->

  - Fixed an unbounded memory leak in `kubectl cnpg logs -f` on busy clusters, where a per-log-group timer was never released; timers are now reused across iterations. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10976](cloudnative-pg/cloudnative-pg#10976)) <!-- 1.29 1.28 1.25 -->
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request Jul 7, 2026
##### [\`v1.30.0\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.0)

**Release date:** Jun 29, 2026

##### Important changes

- Updated the deprecation notice for native (in-tree) Barman Cloud support to reflect that it will now be removed in CloudNativePG 1.31.0, rather than 1.30.0. Users are still encouraged to migrate to the Barman Cloud Plugin. ([#11083](cloudnative-pg/cloudnative-pg#11083)) <!-- 1.29 1.28 -->

- The `cluster` reference is now immutable on the `Database`, `Pooler`, `Publication`, `Subscription`, and `ScheduledBackup` resources. Pointing one of these objects at a different cluster has no well-defined semantics and previously left the controllers in an inconsistent state; the update is now rejected at the API server via a CEL validation rule. ([#10743](cloudnative-pg/cloudnative-pg#10743)) <!-- 1.29 1.28 1.25 -->

##### Features

- **Primary `Lease` for safe primary election**: introduced a Kubernetes `Lease` object (named after the cluster) that acts as a mutex serializing primary promotion: the instance manager must hold the lease before acting as primary and releases it on clean shutdown so replicas can promote without waiting for the full TTL. Timings are configurable via the new `.spec.primaryLease` stanza. The lease is a promotion gate, not a fence. Primary isolation remains responsible for fencing. ([#10627](cloudnative-pg/cloudnative-pg#10627))

- **`DatabaseRole` CRD for declarative role management**: introduced a `DatabaseRole` custom resource that manages a PostgreSQL role as a standalone Kubernetes object, instead of declaring it inline in the `Cluster`'s `.spec.managed.roles` stanza. Each role gets its own lifecycle, status, and RBAC, which suits GitOps workflows and lets role definitions live next to the applications that own them. The spec reuses the same `RoleConfiguration` structure as the inline method, so migrating a role is a matter of moving the stanza into its own manifest. A `databaseRoleReclaimPolicy` field (`retain`, the default, or `delete`) controls what happens to the role when the resource is deleted, mirroring persistent volumes. ([#6155](cloudnative-pg/cloudnative-pg#6155))

- **TLS client certificates for declarative roles**: a `DatabaseRole` can now include a `clientCertificate` block to have the operator automatically generate and renew a TLS client certificate, signed by the cluster's client CA and stored in a `<databaserole-name>-client-cert` Secret. This enables password-free PostgreSQL `cert` authentication; the Secret is cleaned up when the feature is disabled or the `DatabaseRole` is deleted. ([#10896](cloudnative-pg/cloudnative-pg#10896))

- **PgBouncer image management via image catalogs**: the `Pooler` resource can now reference an entry in an `ImageCatalog` or `ClusterImageCatalog` through the new `spec.pgbouncer.imageCatalogRef` field, centralizing PgBouncer image management. When a catalog entry is updated, all referencing `Poolers` are automatically reconciled and roll out the new image without any change to their spec. The resolved image is reported in `status.image`, and a new `status.phase` (`active`, `paused`, `inactive`, or `failed`), also surfaced as a `Phase` column in `kubectl get pooler`, summarizes the lifecycle. ([#10568](cloudnative-pg/cloudnative-pg#10568))

##### Enhancements

- Enabled `pg_upgrade` in-place major upgrades to PostgreSQL 19 or later for clusters that use Image Volume extensions, building on the extension-path support added to `pg_upgrade` in PostgreSQL 19. During the upgrade `Job`, the source- and target-version extension images are mounted side by side, so the old server keeps its libraries and a failed upgrade reverts cleanly. ([#10366](cloudnative-pg/cloudnative-pg#10366)) <!-- 1.29 -->

- Added TLS support for the `Pooler` metrics endpoint via `.spec.monitoring.tls.enabled`. When enabled, the metrics server is served over HTTPS, reusing the certificate and key from `.spec.pgbouncer.clientTLSSecret` and reloading it on every handshake to support rotation without a restart; the generated `PodMonitor` scrapes over `https` accordingly. ([#10466](cloudnative-pg/cloudnative-pg#10466))

- Added a label selector to the `Cluster` scale subresource (`status.selector`), making a `Cluster` a valid `targetRef` for the Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA), which can now map a `Cluster` to its instance pods. Contributed by [@sebv004](https://github.com/sebv004). ([#8996](cloudnative-pg/cloudnative-pg#8996)) <!-- 1.29 1.28 1.25 -->

- The operator now emits a `Warning` `PrimaryStatusCheckFailed` event on the `Cluster` when the primary pod is `Ready` from the kubelet perspective but the operator's `/pg/status` check fails and failover is deferred, giving users visibility into the deferral via `kubectl describe cluster`. ([#10509](cloudnative-pg/cloudnative-pg#10509)) <!-- 1.29 1.28 1.25 -->

- Added the `ENABLE_WEBHOOK_NAMESPACE_SUFFIX` flag, which suffixes the operator's webhook configuration names with `-<OPERATOR_NAMESPACE>` so that multiple operator instances can coexist on the same cluster. The operator only looks up these configurations; users must create and maintain them. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10420](cloudnative-pg/cloudnative-pg#10420)) <!-- 1.29 1.28 1.25 -->

- The operator now reloads a CNPG-i plugin automatically when its pods are rolled: it watches the `EndpointSlices` backing plugin `Services` and re-enqueues every cluster using the plugin once the new pods become `Ready`, so an upgraded plugin is picked up without waiting for the next resync. ([#10836](cloudnative-pg/cloudnative-pg#10836)) <!-- 1.29 1.28 1.25 -->

- Instance serial numbers are now assigned by reusing the lowest free slot among existing instance names, instead of always incrementing a global counter. Pod and PVC names stay stable across instance recreation (for example, an instance recreated after a node drain comes back with the same name), and serials freed by deleted instances are reclaimed. A new `Initialized` cluster condition reports whether the cluster has completed its first bootstrap, and `status.latestGeneratedNode` is deprecated: it is no longer written, but is preserved on the CRD for backward compatibility. ([#10548](cloudnative-pg/cloudnative-pg#10548))

- Defaulting and validation now run during reconciliation as a fallback when admission webhooks are unavailable, or configured to ignore failures, so the operator no longer reconciles invalid or incomplete specs. Missing defaults are applied directly, and validation failures are surfaced in the resource status instead of failing silently later. ([#10874](cloudnative-pg/cloudnative-pg#10874))

##### Security

- **`CVE-2026-55769` / `GHSA-x8c2-3p4r-v9r6`: `search_path` pinning on operator-issued connections**: a database owner could plant overloaded built-in operators in the `public` schema and alter the `search_path` so that operator introspection probes, running as the cluster superuser, resolved those overloads before `pg_catalog`, a `CWE-426` privilege-escalation chain (same class as `CVE-2018-1058`) that could lead to in-pod RCE via `COPY ... FROM PROGRAM`. The operator now pins `search_path = pg_catalog, public, pg_temp` on every pooled connection so it ships in the startup message and takes precedence over tenant-controlled defaults. ([#10774](cloudnative-pg/cloudnative-pg#10774), [`GHSA-x8c2-3p4r-v9r6`](GHSA-x8c2-3p4r-v9r6)) <!-- 1.29 1.28 1.25 -->

- **`GHSA-7qwx-x8ff-3px9`: authenticated operator-to-instance-manager calls**: the instance manager's remote webserver relied on network isolation rather than authentication for its operator-only control endpoints, so any party able to reach the pod's status port could invoke them, disrupting backup orchestration and WAL archival and reading operational metadata. (The upgrade endpoint is SHA-256-pinned, so this did not permit arbitrary code execution.) The operator now generates an in-memory ECDSA P-256 client certificate at startup and reconciles its SHA-256 fingerprint into the cluster status; the instance manager rejects requests to sensitive endpoints that do not present a matching certificate. This hardening is not backported; earlier releases should continue to restrict the status port with a `NetworkPolicy`. ([#10579](cloudnative-pg/cloudnative-pg#10579), [`GHSA-7qwx-x8ff-3px9`](GHSA-7qwx-x8ff-3px9))

- **`CVE-2026-55765` / `GHSA-w3gf-xc94-wvmj`: operator-side SCRAM-SHA-256 password encoding**: the operator now SCRAM-SHA-256 encodes cleartext role passwords before issuing `CREATE`/`ALTER ROLE ... PASSWORD`, so the literal PostgreSQL parses (and that extensions such as `pg_stat_statements` or `pgaudit` may capture) is the SCRAM verifier rather than the cleartext secret. Pre-hashed (MD5 or SCRAM) values are forwarded unchanged, and the per-Secret annotation `cnpg.io/passwordPassthrough: "enabled"` opts out. ([#10724](cloudnative-pg/cloudnative-pg#10724), [`GHSA-w3gf-xc94-wvmj`](GHSA-w3gf-xc94-wvmj)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Added support for Kubernetes 1.36. ([#10900](cloudnative-pg/cloudnative-pg#10900)) <!-- 1.29 1.28 -->

- Updated the default PostgreSQL version to 18.4. ([#10719](cloudnative-pg/cloudnative-pg#10719)) <!-- 1.29 1.28 1.25 -->

- Updated the Kubernetes versions used to test the operator on public cloud providers. ([#10720](cloudnative-pg/cloudnative-pg#10720), <!-- 1.29 1.28 1.25 --> [#10563](cloudnative-pg/cloudnative-pg#10563), <!-- 1.29 1.28 1.25 --> [#11033](cloudnative-pg/cloudnative-pg#11033)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed declarative `Database`, `Publication`, and `Subscription` objects reporting a stale primary-side status forever after their cluster was demoted to a replica; the controller now re-checks the replica condition and watches the `Cluster` so a demotion is detected promptly. ([#10871](cloudnative-pg/cloudnative-pg#10871)) <!-- 1.29 1.28 1.25 -->

- Fixed non-sequential pod names (for example `-1`, `-3`) caused by the instance serial counter being advanced before the corresponding `Job` and PVCs were created; the bump is now persisted only after those resources exist. ([#10491](cloudnative-pg/cloudnative-pg#10491)) <!-- 1.29 1.28 1.25 -->

- Fixed deletion of a `Database`, `Publication`, or `Subscription` getting stuck in `Terminating` on a replica cluster, where the replica gate ran before the finalizer reconciler and the finalizer was never released. On a replica the PostgreSQL object is left to the primary cluster. ([#10853](cloudnative-pg/cloudnative-pg#10853)) <!-- 1.29 1.28 1.25 -->

- Fixed a conflicting duplicate `Database` or `Subscription` with a `delete` reclaim policy dropping the PostgreSQL object owned by the surviving CR; the drop is now gated on a recorded reconciliation. ([#10870](cloudnative-pg/cloudnative-pg#10870)) <!-- 1.29 1.28 1.25 -->

- Fixed the `postgres` superuser being left locked out after superuser access was disabled and then re-enabled, because the cached secret version was not invalidated and the password was never re-applied. Diagnosed by [@mhartmann-jaconi](https://github.com/mhartmann-jaconi). ([#10834](cloudnative-pg/cloudnative-pg#10834)) <!-- 1.29 1.28 1.25 -->

- Fixed backups getting stuck in the `started` phase when the instance manager running them was restarted (for example by the in-place upgrade following an operator upgrade) before the backup reached `running`; the reconciliation is now rescheduled so the lost session is detected. ([#10859](cloudnative-pg/cloudnative-pg#10859)) <!-- 1.29 1.28 -->

- Fixed `exec`/`attach` streaming to negotiate WebSocket with a SPDY fallback, restoring compatibility both with Kubernetes versions that have removed SPDY and with platforms such as OpenShift that reject WebSocket exec upgrades. Contributed by [@bartscheers](https://github.com/bartscheers). ([#10876](cloudnative-pg/cloudnative-pg#10876), [#10933](cloudnative-pg/cloudnative-pg#10933)) <!-- 1.29 -->

- Fixed resource leaks when concurrent `Backup` objects raced: backups now run in strict creation-time order, so an already-executing backup is never preempted by a newer one and its replication slot and PostgreSQL session are no longer orphaned on the primary. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10747](cloudnative-pg/cloudnative-pg#10747)) <!-- 1.29 1.28 1.25 -->

- Fixed role reconciliation clearing the password on a PostgreSQL role when the referenced Secret could not be fetched; the role is now left untouched until the Secret becomes available, and per-action errors are aggregated for better visibility. ([#10053](cloudnative-pg/cloudnative-pg#10053)) <!-- 1.29 1.28 1.25 -->

- Fixed a bootstrap failure where a metrics-exporter setup error (commonly a duplicate-key race with the controller) rolled back `streaming_replica` creation and wedged replica joins. The metrics-exporter step now runs in a separate transaction. Contributed by [@BlaiseAntony](https://github.com/BlaiseAntony). ([#10749](cloudnative-pg/cloudnative-pg#10749)) <!-- 1.29 1.28 1.25 -->

- Fixed a `ScheduledBackup` controller loop that occurred when a `Backup` was created but its status patch never landed; the controller now adopts an existing `Backup` for the next iteration instead of looping on `AlreadyExists`. ([#10612](cloudnative-pg/cloudnative-pg#10612)) <!-- 1.29 1.28 1.25 -->

- Fixed a nil-pointer panic when reconciling a `Pooler` whose `Cluster` has been deleted. ([#10667](cloudnative-pg/cloudnative-pg#10667)) <!-- 1.29 1.28 -->

- Fixed bootstrap log handling so that all named log pipes (`postgres`, `postgres.csv`, and `postgres.json`) get consumers during `WithActiveInstance`, preventing regular files from being created in place of the named pipes. ([#10043](cloudnative-pg/cloudnative-pg#10043)) <!-- 1.29 1.28 1.25 -->

- Fixed generation of invalid IPv6 URLs by wrapping the address in square brackets. Contributed by [@Infinoid](https://github.com/Infinoid). ([#10682](cloudnative-pg/cloudnative-pg#10682)) <!-- 1.29 1.28 1.25 -->

- Fixed an external cluster plugin still being treated as active when its configuration set `enabled: false`. ([#10932](cloudnative-pg/cloudnative-pg#10932)) <!-- 1.29 1.28 1.25 -->

- Fixed a race during bootstrap recovery from an object store where the restore job could read a stale `Cluster` (primary not yet recorded and timeline still unset) and have its `.history` files rejected by the split-brain guard. When this happened, recovery stopped at the base backup's timeline and silently dropped transactions committed on later timelines. History files are now allowed while the cluster timeline is unset. Contributed by [@dennispidun](https://github.com/dennispidun). ([#10818](cloudnative-pg/cloudnative-pg#10818)) <!-- 1.29 1.28 1.25 -->

- Fixed a race where deleting an instance's PVCs could leave the instance permanently stuck: if the data PVC was removed while a WAL PVC was still terminating, the operator recreated the instance bound to the terminating volume, leaving the Pod unschedulable and blocking all further reconciliation. The operator now waits for terminating PVCs to be fully removed before recreating or reattaching an instance, and surfaces the wait through a log line and the cluster phase. ([#11017](cloudnative-pg/cloudnative-pg#11017))

- Fixed a cache race during cluster creation when the server and client CA resolve to the same Secret (the default): a stale informer cache triggered a redundant `Create` that failed with `AlreadyExists` and could leave the cluster stuck in `Unable to create required cluster objects`. The operator now reuses the already-fetched CA Secret when the names match. ([#10989](cloudnative-pg/cloudnative-pg#10989)) <!-- 1.29 1.28 1.25 -->

- Fixed the `pg_basebackup` bootstrap path overwriting or failing on a pre-existing `PGDATA` (for example after a replica Pod restart) by enforcing the same pre-flight directory check already applied by the other bootstrap methods; this also protects statically provisioned PVCs from being silently overwritten. ([#11006](cloudnative-pg/cloudnative-pg#11006)) <!-- 1.29 1.28 1.25 -->

- Fixed the `Cluster` phase flapping between `Healthy` and a plugin-failure phase when a post-reconcile plugin hook returned an error; the `Healthy` phase is now registered as the last step of a successful reconciliation, so a loop that ends in a plugin error never reports `Healthy`. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10421](cloudnative-pg/cloudnative-pg#10421)) <!-- 1.29 1.28 1.25 -->

- Fixed stale certificate data and partial reads after an external server's Secret was rotated (for example a CA bundle shrinking from two certificates to one): the file is now written atomically, so libpq always reads either the old or the new value, never a mix. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10975](cloudnative-pg/cloudnative-pg#10975)) <!-- 1.29 1.28 1.25 -->

- Fixed plugin connectivity to use the plugin `Service` FQDN instead of its short name, avoiding failures when a cluster-level proxy is automatically injected into pods. Contributed by [@kdautrey](https://github.com/kdautrey). ([#10921](cloudnative-pg/cloudnative-pg#10921)) <!-- 1.29 1.28 1.25 -->

- Fixed excessive operator log noise from the per-request `Cluster` create/update validation webhook messages, now logged at `debug` instead of `info`. ([#10984](cloudnative-pg/cloudnative-pg#10984)) <!-- 1.29 1.28 1.25 -->

- Fixed `spec.postgresql.parameters` accepting keys that are not valid PostgreSQL parameter names, which could inject arbitrary directives into `postgresql.conf`; key names are now validated by the webhook. ([#11029](cloudnative-pg/cloudnative-pg#11029)) <!-- 1.29 1.28 1.25 -->

- Fixed a switchover deadlock when a WAL-archiver plugin was enabled on an existing cluster: with `primaryUpdateMethod: switchover` the primary could not be rolled out because a clean demotion needs the archiver sidecar that is still missing. The operator now recreates the primary Pod in place so the sidecar is injected and archiving resumes. The check also covers plugins that inject the archiver as a native sidecar (an init container with `restartPolicy: Always`), such as the Barman Cloud plugin. ([#11032](cloudnative-pg/cloudnative-pg#11032), <!-- 1.29 1.28 1.25 --> [#11059](cloudnative-pg/cloudnative-pg#11059)) <!-- 1.29 1.28 1.25 -->

- Fixed a cluster staying in `Setting up primary` indefinitely when the instance-creation Job exhausted its backoff limit; the operator now detects the terminal Job failure and marks the cluster unrecoverable, naming the failed Job and pointing to its logs. ([#11035](cloudnative-pg/cloudnative-pg#11035)) <!-- 1.29 1.28 1.25 -->

- Fixed a first-primary bootstrap deadlock where a status-patch conflict after the data PVC was created but before the initialization Job was started left the orphan Pending PVC counted as an instance, blocking the bootstrap gate; the PVC-state reconciler now recreates the bootstrap Job reusing the assigned serial. ([#11039](cloudnative-pg/cloudnative-pg#11039)) <!-- 1.29 1.28 1.25 -->

- Fixed external cluster names and secret selector references being joined into filesystem paths without validation, letting a `..` component or path separator escape the external secrets directory when the instance manager dumps connection material; these values are now rejected at the validating webhook and re-checked at the write site. Reported by [@r0binak](https://github.com/r0binak). ([#11045](cloudnative-pg/cloudnative-pg#11045)) <!-- 1.29 1.28 1.25 -->

- Fixed a backup getting stuck in `pending` forever: the concurrent-backup gate ran on every reconcile and could overwrite an already-completed phase written asynchronously by the instance manager. The gate now runs only while the backup phase is still unset or `pending`. ([#11056](cloudnative-pg/cloudnative-pg#11056)) <!-- 1.29 -->

- Fixed a declarative `VolumeSnapshot` backup being permanently marked as failed when a stale cache made the operator re-create a snapshot it had already provisioned, failing with `AlreadyExists`. The operator now tolerates the collision when the existing snapshot carries this backup's label and adopts it; a collision with a foreign snapshot still surfaces as an error. ([#11071](cloudnative-pg/cloudnative-pg#11071)) <!-- 1.29 1.28 1.25 -->

- Fixed a volume snapshot backup being discarded on a transient instance-manager connection error (for example a dial timeout from a brief pod-network disruption) during the finalize step, even when its snapshots were already provisioned; such network errors are now retried instead of treated as terminal. ([#11069](cloudnative-pg/cloudnative-pg#11069)) <!-- 1.29 1.28 1.25 -->

- Fixed a replica switchover losing its `status.demotionToken` when a reconcile was requeued between storing the token and cleaning up the transition metadata (for example a cleanup patch failing against a flaky webhook); the empty no-change token is no longer patched back over the stored value. ([#11075](cloudnative-pg/cloudnative-pg#11075)) <!-- 1.29 1.28 1.25 -->

- `cnpg` plugin:

  - Fixed `kubectl cnpg psql` on Windows, where execution relied on a Unix-only system call and failed with "not supported by windows"; Windows now launches `kubectl exec` as a child process. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#10972](cloudnative-pg/cloudnative-pg#10972)) <!-- 1.29 1.28 1.25 -->

  - Fixed an unbounded memory leak in `kubectl cnpg logs -f` on busy clusters, where a per-log-group timer was never released; timers are now reused across iterations. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10976](cloudnative-pg/cloudnative-pg#10976)) <!-- 1.29 1.28 1.25 -->

##### Supported versions

- Kubernetes 1.36, 1.35, and 1.34
- PostgreSQL 18, 17, 16, 15, and 14
  - PostgreSQL 18.4 is the default image
  - PostgreSQL 14 support ends on November 12, 2026

---
##### [\`v1.29.2\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.2)

**Release date:** Jun 29, 2026

##### Important changes

- Updated the deprecation notice for native (in-tree) Barman Cloud support to reflect that it will now be removed in CloudNativePG 1.31.0, rather than 1.30.0. Users are still encouraged to migrate to the Barman Cloud Plugin. ([#11083](cloudnative-pg/cloudnative-pg#11083)) <!-- 1.29 1.28 -->

- The `cluster` reference is now immutable on the `Database`, `Pooler`, `Publication`, `Subscription`, and `ScheduledBackup` resources. Pointing one of these objects at a different cluster has no well-defined semantics and previously left the controllers in an inconsistent state; the update is now rejected at the API server via a CEL validation rule. ([#10743](cloudnative-pg/cloudnative-pg#10743)) <!-- 1.29 1.28 1.25 -->

##### Enhancements

- Enabled `pg_upgrade` in-place major upgrades to PostgreSQL 19 or later for clusters that use Image Volume extensions, building on the extension-path support added to `pg_upgrade` in PostgreSQL 19. During the upgrade `Job`, the source- and target-version extension images are mounted side by side, so the old server keeps its libraries and a failed upgrade reverts cleanly. ([#10366](cloudnative-pg/cloudnative-pg#10366)) <!-- 1.29 -->

- Added a label selector to the `Cluster` scale subresource (`status.selector`), making a `Cluster` a valid `targetRef` for the Vertical Pod Autoscaler (VPA) and Horizontal Pod Autoscaler (HPA), which can now map a `Cluster` to its instance pods. Contributed by [@sebv004](https://github.com/sebv004). ([#8996](cloudnative-pg/cloudnative-pg#8996)) <!-- 1.29 1.28 1.25 -->

- The operator now emits a `Warning` `PrimaryStatusCheckFailed` event on the `Cluster` when the primary pod is `Ready` from the kubelet perspective but the operator's `/pg/status` check fails and failover is deferred, giving users visibility into the deferral via `kubectl describe cluster`. ([#10509](cloudnative-pg/cloudnative-pg#10509)) <!-- 1.29 1.28 1.25 -->

- The operator now reloads a CNPG-i plugin automatically when its pods are rolled: it watches the `EndpointSlices` backing plugin `Services` and re-enqueues every cluster using the plugin once the new pods become `Ready`, so an upgraded plugin is picked up without waiting for the next resync. ([#10836](cloudnative-pg/cloudnative-pg#10836)) <!-- 1.29 1.28 1.25 -->

##### Security and Supply Chain

- **`CVE-2026-55769` / `GHSA-x8c2-3p4r-v9r6`: `search_path` pinning on operator-issued connections**: a database owner could plant overloaded built-in operators in the `public` schema and alter the `search_path` so that operator introspection probes, running as the cluster superuser, resolved those overloads before `pg_catalog`, a `CWE-426` privilege-escalation chain (same class as `CVE-2018-1058`) that could lead to in-pod RCE via `COPY ... FROM PROGRAM`. The operator now pins `search_path = pg_catalog, public, pg_temp` on every pooled connection so it ships in the startup message and takes precedence over tenant-controlled defaults. ([#10774](cloudnative-pg/cloudnative-pg#10774), [`GHSA-x8c2-3p4r-v9r6`](GHSA-x8c2-3p4r-v9r6)) <!-- 1.29 1.28 1.25 -->

- **`CVE-2026-55765` / `GHSA-w3gf-xc94-wvmj`: operator-side SCRAM-SHA-256 password encoding**: the operator now SCRAM-SHA-256 encodes cleartext role passwords before issuing `CREATE`/`ALTER ROLE ... PASSWORD`, so the literal PostgreSQL parses (and that extensions such as `pg_stat_statements` or `pgaudit` may capture) is the SCRAM verifier rather than the cleartext secret. Pre-hashed (MD5 or SCRAM) values are forwarded unchanged, and the per-Secret annotation `cnpg.io/passwordPassthrough: "enabled"` opts out. ([#10724](cloudnative-pg/cloudnative-pg#10724), [`GHSA-w3gf-xc94-wvmj`](GHSA-w3gf-xc94-wvmj)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Added support for Kubernetes 1.36. ([#10900](cloudnative-pg/cloudnative-pg#10900)) <!-- 1.29 1.28 -->

- Updated the default PostgreSQL version to 18.4. ([#10719](cloudnative-pg/cloudnative-pg#10719)) <!-- 1.29 1.28 1.25 -->

- Updated the Kubernetes versions used to test the operator on public cloud providers. ([#10720](cloudnative-pg/cloudnative-pg#10720), <!-- 1.29 1.28 1.25 --> [#10563](cloudnative-pg/cloudnative-pg#10563), <!-- 1.29 1.28 1.25 --> [#11033](cloudnative-pg/cloudnative-pg#11033)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed `spec.postgresql.parameters` accepting keys that are not valid PostgreSQL parameter names, which could inject arbitrary directives into `postgresql.conf`; key names are now validated by the webhook. ([#11029](cloudnative-pg/cloudnative-pg#11029)) <!-- 1.29 1.28 1.25 -->

- Fixed declarative `Database`, `Publication`, and `Subscription` objects reporting a stale primary-side status forever after their cluster was demoted to a replica; the controller now re-checks the replica condition and watches the `Cluster` so a demotion is detected promptly. ([#10871](cloudnative-pg/cloudnative-pg#10871)) <!-- 1.29 1.28 1.25 -->

- Fixed non-sequential pod names (for example `-1`, `-3`) caused by the instance serial counter being advanced before the corresponding `Job` and PVCs were created; the bump is now persisted only after those resources exist. ([#10491](cloudnative-pg/cloudnative-pg#10491)) <!-- 1.29 1.28 1.25 -->

- Fixed a switchover deadlock when a WAL-archiver plugin was enabled on an existing cluster: with `primaryUpdateMethod: switchover` the primary could not be rolled out because a clean demotion needs the archiver sidecar that is still missing. The operator now recreates the primary Pod in place so the sidecar is injected and archiving resumes. The check also covers plugins that inject the archiver as a native sidecar (an init container with `restartPolicy: Always`), such as the Barman Cloud plugin. ([#11032](cloudnative-pg/cloudnative-pg#11032), <!-- 1.29 1.28 1.25 --> [#11059](cloudnative-pg/cloudnative-pg#11059)) <!-- 1.29 1.28 1.25 -->

- Fixed a cluster staying in `Setting up primary` indefinitely when the instance-creation Job exhausted its backoff limit; the operator now detects the terminal Job failure and marks the cluster unrecoverable, naming the failed Job and pointing to its logs. ([#11035](cloudnative-pg/cloudnative-pg#11035)) <!-- 1.29 1.28 1.25 -->

- Fixed deletion of a `Database`, `Publication`, or `Subscription` getting stuck in `Terminating` on a replica cluster, where the replica gate ran before the finalizer reconciler and the finalizer was never released. On a replica the PostgreSQL object is left to the primary cluster. ([#10853](cloudnative-pg/cloudnative-pg#10853)) <!-- 1.29 1.28 1.25 -->

- Fixed a conflicting duplicate `Database` or `Subscription` with a `delete` reclaim policy dropping the PostgreSQL object owned by the surviving CR; the drop is now gated on a recorded reconciliation. ([#10870](cloudnative-pg/cloudnative-pg#10870)) <!-- 1.29 1.28 1.25 -->

- Fixed the `postgres` superuser being left locked out after superuser access was disabled and then re-enabled, because the cached secret version was not invalidated and the password was never re-applied. Diagnosed by [@mhartmann-jaconi](https://github.com/mhartmann-jaconi). ([#10834](cloudnative-pg/cloudnative-pg#10834)) <!-- 1.29 1.28 1.25 -->

- Fixed backups getting stuck in the `started` phase when the instance manager running them was restarted (for example by the in-place upgrade following an operator upgrade) before the backup reached `running`; the reconciliation is now rescheduled so the lost session is detected. ([#10859](cloudnative-pg/cloudnative-pg#10859)) <!-- 1.29 1.28 -->

- Fixed `exec`/`attach` streaming to negotiate WebSocket with a SPDY fallback, restoring compatibility both with Kubernetes versions that have removed SPDY and with platforms such as OpenShift that reject WebSocket exec upgrades. Contributed by [@bartscheers](https://github.com/bartscheers). ([#10876](cloudnative-pg/cloudnative-pg#10876), [#10933](cloudnative-pg/cloudnative-pg#10933)) <!-- 1.29 -->

- Fixed resource leaks when concurrent `Backup` objects raced: backups now run in strict creation-time order, so an already-executing backup is never preempted by a newer one and its replication slot and PostgreSQL session are no longer orphaned on the primary. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10747](cloudnative-pg/cloudnative-pg#10747)) <!-- 1.29 1.28 1.25 -->

- Fixed role reconciliation clearing the password on a PostgreSQL role when the referenced Secret could not be fetched; the role is now left untouched until the Secret becomes available, and per-action errors are aggregated for better visibility. ([#10053](cloudnative-pg/cloudnative-pg#10053)) <!-- 1.29 1.28 1.25 -->

- Fixed a bootstrap failure where a metrics-exporter setup error (commonly a duplicate-key race with the controller) rolled back `streaming_replica` creation and wedged replica joins. The metrics-exporter step now runs in a separate transaction. Contributed by [@BlaiseAntony](https://github.com/BlaiseAntony). ([#10749](cloudnative-pg/cloudnative-pg#10749)) <!-- 1.29 1.28 1.25 -->

- Fixed a `ScheduledBackup` controller loop that occurred when a `Backup` was created but its status patch never landed; the controller now adopts an existing `Backup` for the next iteration instead of looping on `AlreadyExists`. ([#10612](cloudnative-pg/cloudnative-pg#10612)) <!-- 1.29 1.28 1.25 -->

- Fixed a nil-pointer panic when reconciling a `Pooler` whose `Cluster` has been deleted. ([#10667](cloudnative-pg/cloudnative-pg#10667)) <!-- 1.29 1.28 -->

- Fixed bootstrap log handling so that all named log pipes (`postgres`, `postgres.csv`, and `postgres.json`) get consumers during `WithActiveInstance`, preventing regular files from being created in place of the named pipes. ([#10043](cloudnative-pg/cloudnative-pg#10043)) <!-- 1.29 1.28 1.25 -->

- Fixed generation of invalid IPv6 URLs by wrapping the address in square brackets. Contributed by [@Infinoid](https://github.com/Infinoid). ([#10682](cloudnative-pg/cloudnative-pg#10682)) <!-- 1.29 1.28 1.25 -->

- Fixed an external cluster plugin still being treated as active when its configuration set `enabled: false`. ([#10932](cloudnative-pg/cloudnative-pg#10932)) <!-- 1.29 1.28 1.25 -->

- Fixed a race during bootstrap recovery from an object store where the restore job could read a stale `Cluster` (primary not yet recorded and timeline still unset) and have its `.history` files rejected by the split-brain guard. When this happened, recovery stopped at the base backup's timeline and silently dropped transactions committed on later timelines. History files are now allowed while the cluster timeline is unset. Contributed by [@dennispidun](https://github.com/dennispidun). ([#10818](cloudnative-pg/cloudnative-pg#10818)) <!-- 1.29 1.28 1.25 -->

- Fixed a cache race during cluster creation when the server and client CA resolve to the same Secret (the default): a stale informer cache triggered a redundant `Create` that failed with `AlreadyExists` and could leave the cluster stuck in `Unable to create required cluster objects`. The operator now reuses the already-fetched CA Secret when the names match. ([#10989](cloudnative-pg/cloudnative-pg#10989)) <!-- 1.29 1.28 1.25 -->

- Fixed the `pg_basebackup` bootstrap path overwriting or failing on a pre-existing `PGDATA` (for example after a replica Pod restart) by enforcing the same pre-flight directory check already applied by the other bootstrap methods; this also protects statically provisioned PVCs from being silently overwritten. ([#11006](cloudnative-pg/cloudnative-pg#11006)) <!-- 1.29 1.28 1.25 -->

- Fixed the `Cluster` phase flapping between `Healthy` and a plugin-failure phase when a post-reconcile plugin hook returned an error; the `Healthy` phase is now registered as the last step of a successful reconciliation, so a loop that ends in a plugin error never reports `Healthy`. Contributed by [@GabriFedi97](https://github.com/GabriFedi97). ([#10421](cloudnative-pg/cloudnative-pg#10421)) <!-- 1.29 1.28 1.25 -->

- Fixed stale certificate data and partial reads after an external server's Secret was rotated (for example a CA bundle shrinking from two certificates to one): the file is now written atomically, so libpq always reads either the old or the new value, never a mix. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10975](cloudnative-pg/cloudnative-pg#10975)) <!-- 1.29 1.28 1.25 -->

- Fixed plugin connectivity to use the plugin `Service` FQDN instead of its short name, avoiding failures when a cluster-level proxy is automatically injected into pods. Contributed by [@kdautrey](https://github.com/kdautrey). ([#10921](cloudnative-pg/cloudnative-pg#10921)) <!-- 1.29 1.28 1.25 -->

- Fixed excessive operator log noise from the per-request `Cluster` create/update validation webhook messages, now logged at `debug` instead of `info`. ([#10984](cloudnative-pg/cloudnative-pg#10984)) <!-- 1.29 1.28 1.25 -->

- Fixed a first-primary bootstrap deadlock where a status-patch conflict after the data PVC was created but before the initialization Job was started left the orphan Pending PVC counted as an instance, blocking the bootstrap gate; the PVC-state reconciler now recreates the bootstrap Job reusing the assigned serial. ([#11039](cloudnative-pg/cloudnative-pg#11039)) <!-- 1.29 1.28 1.25 -->

- Fixed external cluster names and secret selector references being joined into filesystem paths without validation, letting a `..` component or path separator escape the external secrets directory when the instance manager dumps connection material; these values are now rejected at the validating webhook and re-checked at the write site. Reported by [@r0binak](https://github.com/r0binak). ([#11045](cloudnative-pg/cloudnative-pg#11045)) <!-- 1.29 1.28 1.25 -->

- Fixed a backup getting stuck in `pending` forever: the concurrent-backup gate ran on every reconcile and could overwrite an already-completed phase written asynchronously by the instance manager. The gate now runs only while the backup phase is still unset or `pending`. ([#11056](cloudnative-pg/cloudnative-pg#11056)) <!-- 1.29 -->

- Fixed a declarative `VolumeSnapshot` backup being permanently marked as failed when a stale cache made the operator re-create a snapshot it had already provisioned, failing with `AlreadyExists`. The operator now tolerates the collision when the existing snapshot carries this backup's label and adopts it; a collision with a foreign snapshot still surfaces as an error. ([#11071](cloudnative-pg/cloudnative-pg#11071)) <!-- 1.29 1.28 1.25 -->

- Fixed a volume snapshot backup being discarded on a transient instance-manager connection error (for example a dial timeout from a brief pod-network disruption) during the finalize step, even when its snapshots were already provisioned; such network errors are now retried instead of treated as terminal. ([#11069](cloudnative-pg/cloudnative-pg#11069)) <!-- 1.29 1.28 1.25 -->

- Fixed a replica switchover losing its `status.demotionToken` when a reconcile was requeued between storing the token and cleaning up the transition metadata (for example a cleanup patch failing against a flaky webhook); the empty no-change token is no longer patched back over the stored value. ([#11075](cloudnative-pg/cloudnative-pg#11075)) <!-- 1.29 1.28 1.25 -->

- `cnpg` plugin:

  - Fixed `kubectl cnpg psql` on Windows, where execution relied on a Unix-only system call and failed with "not supported by windows"; Windows now launches `kubectl exec` as a child process. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#10972](cloudnative-pg/cloudnative-pg#10972)) <!-- 1.29 1.28 1.25 -->

  - Fixed an unbounded memory leak in `kubectl cnpg logs -f` on busy clusters, where a per-log-group timer was never released; timers are now reused across iterations. Contributed by [@Anand-240](https://github.com/Anand-240). ([#10976](cloudnative-pg/cloudnative-pg#10976)) <!-- 1.29 1.28 1.25 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Backup election logic can preempt an already-started backup

4 participants