Skip to content

backups: a restore and a backup of the same application are not mutually exclusive #4559

Description

@mattia-eleuteri

Describe the bug

Nothing coordinates a RestoreJob with a BackupJob of the same application. #4544 makes a BackupJob that has not started yet (status.startedAt unset) wait while a non-terminal RestoreJob writes into its application, but it leaves two cases open, both outside what a lookup at the BackupJob gate can close:

  1. A restore that starts against a running backup. Once a driver has stamped startedAt, the gate no longer looks at restores, and the RestoreJob controller does not look at backups. On the CNPG path every BackupJob reconcile re-applies the Cluster's barman plugin, so a running backup can re-point an in-place restored cluster at the source's WAL prefix, the failure fix(backups): hold a BackupJob while its application is being restored #4544 describes. The Job strategy stamps startedAt, requeues after 5s and creates its Job on the next pass, so a RestoreJob created in between runs alongside a backup Job that has not started moving data yet.
  2. A RestoreJob created between the gate's List of RestoreJobs, which reads the informer cache, and the driver stamping startedAt. The window is short, but the cache can lag a RestoreJob created just before.

Environment

To Reproduce

  1. Create a BackupJob for a Postgres application and let the CNPG driver stamp status.startedAt.
  2. While the backup runs, create an in-place RestoreJob for the same application.
  3. The restore starts at once, and BackupJob reconciles keep re-applying the barman plugin to the Cluster the restore is re-rendering.

Expected behaviour
A backup and a restore of the same application never run at the same time: a RestoreJob waits while a BackupJob of its target application is running, as a BackupJob now waits for a RestoreJob, and the two decisions do not race each other.

Actual behaviour
The RestoreJob starts regardless of a running BackupJob, and the BackupJob gate only covers restores that exist, and are visible in the cache, before the backup starts.

Additional context
Checking each side from the other still leaves a race when both reconcile at the same moment. Closing it needs a claim per application that both controllers take atomically before starting work, and that every driver respects: for instance a coordination.k8s.io Lease named after the application, or an annotation on the application written with a resourceVersion precondition, with the loser requeuing. That touches the RestoreJob controller and all strategy drivers, which is why it was left out of #4544. Raised in review there: #4544 (review)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)kind/bugCategorizes issue or PR as related to a bugtriage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions