Skip to content

[Bug]: postgres-operator on release-1.4/1.5 pins CNPG 1.27.3 over 1.27.1 CRDs, causing false backup failures #3479

Closed

Description

@IvanHunters

Summary

On release-1.4 (v1.4.6) and release-1.5 (v1.5.3), packages/system/postgres-operator vendors the cloudnative-pg chart 0.26.1 (CRDs from CNPG 1.27.1) but pins image.tag: "1.27.3". Operator 1.27.3 writes status.instanceID.sessionID to the Backup status (field added in 1.27.3). The 1.27.1 backups CRD has no such field, so the structural schema prunes it on write. The operator then reads an empty storedSessionID, takes the isInstanceManagerRestarted fail-safe branch, and marks every backup as failed with event InstanceManagerRestarted and error instance manager was restarted during backup, even though the backup physically completes and the recovery window is correct.

Affected

v1.4.6, v1.5.3, and current HEAD of release-1.4 / release-1.5 (PR #3313 was not backported).

Impact (mostly status/metrics, with caveats)

  • cluster.status.lastFailedBackup is set after every successful backup; lastSuccessfulBackup is never set.
  • Legacy metric cnpg_collector_last_failed_backup_timestamp reports a failure after every successful backup, causing false alerts.
  • False-failed Backup objects are never garbage-collected (GC only collects Completed), so they accumulate in etcd.
  • A new cluster restoring via spec.bootstrap.recovery.backup.name pointing at such an object blocks forever (checkReadyForRecovery). Restoring from the object store directly is unaffected.
  • Retention, recovery window, firstRecoverabilityPoint, and failover/switchover are not affected.

Fixed in

1.6.0 via #3313, but that PR migrates backups to the barman-cloud plugin (the fix for the standard-image-variant issue #3300) and is too large to backport wholesale to a patch branch.

Suggested minimal fix for release-1.4 / release-1.5

Drop the image.tag: "1.27.3" pin in packages/system/postgres-operator/values.yaml. The vendored chart defaults image.tag to its appVersion (1.27.1), which matches the shipped CRDs, removing the skew. The image variant on 1.4/1.5 still bundles the barman-cloud binaries, so physical backups keep working.

Origin

Introduced by commit 845192b9 ("Update to v1.27.3"), which downgraded the vendored chart to 0.26.1/1.27.1 and simultaneously pinned the operator image to 1.27.3.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    triage/not-reproducibleIndicates an issue can not be reproduced as described
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  2. lexfrei commented on Sep 23, 2026

    @lexfrei
    Contributor

    Fixed by #3526 (release-1.5) and #3528 (release-1.4): the vendored CNPG CRDs match the 1.27.3 operator now, so sessionID is no longer pruned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugtriage/not-reproducibleIndicates an issue can not be reproduced as described

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions