You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
backups: a restore and a backup of the same application are not mutually exclusive #4559
Nothing coordinates a RestoreJob with a BackupJob of the same application. #4544 makes a BackupJob that has not started yet (status.startedAt unset) wait while a non-terminal RestoreJob writes into its application, but it leaves two cases open, both outside what a lookup at the BackupJob gate can close:
A restore that starts against a running backup. Once a driver has stamped startedAt, the gate no longer looks at restores, and the RestoreJob controller does not look at backups. On the CNPG path every BackupJob reconcile re-applies the Cluster's barman plugin, so a running backup can re-point an in-place restored cluster at the source's WAL prefix, the failure fix(backups): hold a BackupJob while its application is being restored #4544 describes. The Job strategy stamps startedAt, requeues after 5s and creates its Job on the next pass, so a RestoreJob created in between runs alongside a backup Job that has not started moving data yet.
A RestoreJob created between the gate's List of RestoreJobs, which reads the informer cache, and the driver stamping startedAt. The window is short, but the cache can lag a RestoreJob created just before.
Create a BackupJob for a Postgres application and let the CNPG driver stamp status.startedAt.
While the backup runs, create an in-place RestoreJob for the same application.
The restore starts at once, and BackupJob reconciles keep re-applying the barman plugin to the Cluster the restore is re-rendering.
Expected behaviour
A backup and a restore of the same application never run at the same time: a RestoreJob waits while a BackupJob of its target application is running, as a BackupJob now waits for a RestoreJob, and the two decisions do not race each other.
Actual behaviour
The RestoreJob starts regardless of a running BackupJob, and the BackupJob gate only covers restores that exist, and are visible in the cache, before the backup starts.
Additional context
Checking each side from the other still leaves a race when both reconcile at the same moment. Closing it needs a claim per application that both controllers take atomically before starting work, and that every driver respects: for instance a coordination.k8s.io Lease named after the application, or an annotation on the application written with a resourceVersion precondition, with the loser requeuing. That touches the RestoreJob controller and all strategy drivers, which is why it was left out of #4544. Raised in review there: #4544 (review)
Describe the bug
Nothing coordinates a RestoreJob with a BackupJob of the same application. #4544 makes a BackupJob that has not started yet (
status.startedAtunset) wait while a non-terminal RestoreJob writes into its application, but it leaves two cases open, both outside what a lookup at the BackupJob gate can close:startedAt, the gate no longer looks at restores, and the RestoreJob controller does not look at backups. On the CNPG path every BackupJob reconcile re-applies the Cluster's barman plugin, so a running backup can re-point an in-place restored cluster at the source's WAL prefix, the failure fix(backups): hold a BackupJob while its application is being restored #4544 describes. The Job strategy stampsstartedAt, requeues after 5s and creates its Job on the next pass, so a RestoreJob created in between runs alongside a backup Job that has not started moving data yet.Listof RestoreJobs, which reads the informer cache, and the driver stampingstartedAt. The window is short, but the cache can lag a RestoreJob created just before.Environment
mainafter fix(backups): hold a BackupJob while its application is being restored #4544; the gate is also a backport candidate forrelease-1.6To Reproduce
status.startedAt.Expected behaviour
A backup and a restore of the same application never run at the same time: a RestoreJob waits while a BackupJob of its target application is running, as a BackupJob now waits for a RestoreJob, and the two decisions do not race each other.
Actual behaviour
The RestoreJob starts regardless of a running BackupJob, and the BackupJob gate only covers restores that exist, and are visible in the cache, before the backup starts.
Additional context
Checking each side from the other still leaves a race when both reconcile at the same moment. Closing it needs a claim per application that both controllers take atomically before starting work, and that every driver respects: for instance a
coordination.k8s.ioLease named after the application, or an annotation on the application written with aresourceVersionprecondition, with the loser requeuing. That touches the RestoreJob controller and all strategy drivers, which is why it was left out of #4544. Raised in review there: #4544 (review)