Skip to content

[Backport release-1.6] fix(postgres): let in-place restore re-bootstrap instead of wedging - #3988

Merged
myasnikovdaniil merged 1 commit into
release-1.6from
backport-3959-to-release-1.6
Sep 3, 2026
Merged

myasnikovdaniil merged 1 commit into
release-1.6from
backport-3959-to-release-1.6

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3959 to release-1.6.

@github-actions

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3959-to-release-1.6
git worktree add --checkout .worktree/backport-3959-to-release-1.6 backport-3959-to-release-1.6
cd .worktree/backport-3959-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x 5bcdb8cfb8d694d20e7e8891fbc38f2910f24293
git push --force-with-lease

An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
(cherry picked from commit 5bcdb8c)
Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: LLM
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-3959-to-release-1.6 branch from 25cab21 to 22b1e07 Compare September 2, 2026 15:35
@github-actions github-actions Bot added area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Sep 2, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review September 3, 2026 05:38
@myasnikovdaniil
myasnikovdaniil merged commit b158962 into release-1.6 Sep 3, 2026
72 of 74 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3959-to-release-1.6 branch September 3, 2026 07:27
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Sep 29, 2026
…efix (#4299)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #4298.

`applyClusterPluginBackup` preserves the `serverName` already on the
Cluster so that a backup does not flip the WAL-archive prefix
mid-stream, which would split the archive across two prefixes. That rule
is right; the read it rests on is not reliable in one window.

The value was read through the manager's cached client, and
`BackupJobReconciler.SetupWithManager` registers `For(&BackupJob{})` and
nothing else, so a Cluster only reaches the cache through a lazily
started informer with no resync tied to the Cluster's own changes. An
in-place `RestoreJob` deletes the Cluster and the chart re-renders it
with `bootstrap.newServerName` seconds later, so a read served during
that window describes the object that was purged — and the source's
`serverName` is written back over the freshly rendered one.
`barman-cloud-check-wal-archive` then refuses the destination with
`Expected empty archive` and the `RestoreJob` ends on its 30-minute
deadline, which is the failure `newServerName` was added in #3988 to
prevent.

The same stale read carries a stale `UID`, and that `UID` owns the
`ObjectStore` this function applies, so the store is garbage-collected
as soon as it is written.

Reading through the dynamic client this reconciler already carries keeps
the preservation rule and makes the value it rests on truthful. There is
precedent in the same file: the HelmRelease suspend/resume path reads
that way for the same reason.

Observed on a v1.6.3 cluster, twice on separate runs with the same
four-second gap between the chart's apply and the driver's. The
regression test makes the two clients disagree the way they disagree in
that window and asserts the driver believes the live one; reverting the
fix fails it on the serverName assertion.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): an in-place restore no longer loses its fresh WAL-archive prefix to a stale Cluster read, which left the restored cluster archiving onto the source's prefix and the RestoreJob failing on its deadline
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Backup processing now uses the latest CNPG Cluster state rather than
potentially outdated cached information. This ensures the cluster’s
current server name and identity are used when associating backup
ObjectStore resources, including when the cached and live cluster
details differ.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants