Repository navigation
linstor: restoring a PVC from a VolumeSnapshot leaves two of three replicas empty (silent data loss) #4862
Description
Activity
I went through the LINSTOR v1.33.3 and linstor-csi v1.11.2 sources. The empty replicas come from two separate problems, and the resource-group workaround from the description would not fix either of them for snapshots that already exist.
The first problem is a stale winner of the initial UpToDate race. linstor-csi creates the volume definition with
RestoreVolumeDefinitionSnapshot, which copies onlyDrbdCurrentGi, so the new volume definition has noDRBD_INITIALIZEDflag. The restore then copies the snapshot's volume definition properties into it, andLinstor/Drbd/InitialUpToDateOnstill names the node that won the race for the source volume (seerestoreOnNode). The restored replica already has metadata, so it never takes part in the race. When autoplace puts a replica on that old winner node,DrbdLayer.createMetaDatagives the empty zvol a random current UUID and marks it UpToDate. This matches the run whereInitialUpToDateOnwasNODE0and the restore ran onnode1. Since the empty volume has no filesystem, the winner path also runs mkfs on it.The second problem is the
skipInitSyncshortcut for the other autoplaced replicas, as described in the report.DrbdOptions/ForceInitialSyncon the resource group or the controller does not help here.copyForceInitialSyncPropruns increateResourceonly, not increateResourceFromSnapshot. So the restored resource definition getsForceInitialSyncPermanentwhen autoplace runs, aftercheckDrbdInitializedStatehas already found nothing to do, and the stale winner stays. It would only cover snapshots of volumes created after the property was set, and every new thick ZFS volume would pay a full initial sync for that.LINSTOR already does the right thing for backup shipping.
CtrlBackupCreateApiCallHandlersetsForceInitialSyncPermanenton the resource definition props of the snapshot. On restore this recreates metadata on the restored replica, makes it the race winner and clearsDRBD_INITIALIZED, so every other replica does a full sync from it. The existing mkfs step checks for a filesystem first, so restored data is not touched. My plan is to set the same property on the target resource definition at the start of a restore that comes through the API. Rollback goes through the same function but restores every node from its own snapshot, so it stays as is. The cost lands only on restored volumes: one full sync per extra replica, which a one-node restore can't avoid anyway. New volumes keep skipping the initial sync.The stale winner path does not depend on the storage provider, so LVM thick is affected too, and thin pools hit
skipInitSyncthe same way as ZFS.DrbdOptions/auto-diskfulis enabled in Cozystack and adds diskful replicas later throughcreateResource, so a restored volume can get an empty replica long after the restore.In Cozystack this hits every PVC created from a VolumeSnapshot. That includes Velero backups with
snapshotMoveData: the data mover reads a PVC restored from the snapshot, so the backup itself can capture an empty volume. Tenant Kubernetes PVCs restored through kubevirt-csi go the same way, via a CDI DataVolume with a snapshot source. Golden image clones usecsi-clone, and LINSTOR clones on every source node and copies the volume definition flags, so that path looks fine. I'll check it on a test cluster together with the fix.Next: a piraeus-server patch for the controller, a check on a three-node cluster (checksum from every replica,
drbdadm verifywith zero out-of-sync, a snapshot taken before the change), resync time, and the same change sent upstream.A correction to my previous comment first. Only the stale-winner problem needs a fix, and LINBIT already fixed it upstream in LINBIT/linstor-server@9d8cdf5, released in v1.34.0. I backported that commit to our v1.33.3 build and ran it on a three-node cluster. The replicas that skip the initial sync are fine in practice. DRBD resyncs them from the restored replica, and the backing zvols end up byte-identical. A carried patch that forces a full initial sync is not needed, so I dropped that plan.
Results with the backport: 5 restores out of 5 came up with every replica connected and UpToDate, the same checksum from a pod on each node, and identical zvols on all three nodes. In 4 of the 5, the source volume's winner was on a different node than the restore, which is the case that broke before. A snapshot taken before the change restores fine too. The resync after a restore costs about the written data: around 40 s for 1Gi and 134 s for 7.5 GiB written on a 10Gi volume.
One measurement trap, in case someone repeats this: compare replicas through their backing devices only with
dd iflag=direct. A buffered read of/dev/zvol/...can return stale page cache. On the same replica I got zeros from a buffered read and the real data withO_DIRECT.Known limitation, which upstream v1.34 shares since that code did not change: the autoplaced replicas rely on the source volume's day0 UUID being in the restored replica's history. If it ever isn't, DRBD goes StandAlone with "unrelated data" instead of syncing. That failure is visible, not silent, and I could not reproduce it.
How to check and repair volumes restored before the fix
The fix does not touch volumes that were already restored. In every broken restore I saw, exactly one replica was wrong: the stale winner. It holds an empty filesystem created by LINSTOR, while the restored replica and the others are intact. Every replica still shows UpToDate. The only sign in
linstor resource listisStandAlonein the connections column, and nothing marks the volume as degraded.An affected volume has all of these:
Linstor/Drbd/InitialUptoDateOninlinstor volume-definition list-properties <pv> 0names a node whose backing zvol has no origin (zfs get -H -o value origin <pool>/<pv>_00000prints-), while another node's zvol has a snapshot as its origin.- That replica is
StandAlonetowards every peer inlinstor resource list --resources <pv>. - The kernel log on that node has
uuid_compare()=unrelated-dataandUnrelated data, aborting!for the resource.
On a healthy restored volume,
InitialUptoDateOnnames the node whose zvol is a clone of the snapshot, and every connection is Ok.To repair it, recreate the bad replica so it resyncs from the others:
linstor resource delete <stale-node> <pv> linstor resource create <stale-node> <pv> --storage-pool <pool>
drbdadm connect --discard-my-datadoes not help here, because DRBD treats unrelated data differently from a split brain. I tested the delete and recreate on a broken volume: the replica resynced and its zvol matched the other two byte for byte.Before deleting, check where the volume was attached. If an application already ran on the stale replica, it wrote to a different filesystem than the one on the other replicas. Deleting the replica throws those writes away, so decide which copy to keep first.
- added 4 commits that reference this issue
on Oct 11, 2026
Describe the bug
Restoring a PVC from a LINSTOR
VolumeSnapshotgives one replica with the data and two empty replicas, and all three are marked UpToDate. A pod that lands on an empty replica mounts a filesystem without the snapshot's files. This is silent data loss on restore: nothing fails up front, and a reader can see an empty volume before anything looks wrong.Environment
1d813c944, plus the same main with thepiraeus-serverimage rebuilt on Debian trixie (an open dependency roll-up). Both behave the same, so the base image is not the cause.data.replicatedwithplacementCount: "3",storagePool: data,allowRemoteVolumeAccess: "true".To Reproduce
replicated, write a file from a pod, record its sha256, delete the pod.VolumeSnapshotwith classlinstor-snapshotsand wait forreadyToUse.dataSourceset to the snapshot and a pod that reads the file.zfs get origin data/<pv>_00000on each node andlinstor resource list --resources <pv>.Expected behaviour
All three replicas hold the snapshot's data, as clones or after a full resync from the restored replica, and the pod reads the same sha256.
Actual behaviour
Five restores out of five ended the same way, on both satellite images.
Only one node's zvol is a clone of the snapshot. On the other two nodes
originis-, so these are new empty zvols. The satellite on those nodes logsDRBD meta data created for <pv>/0and thenDRBD skipping initial sync for <pv>/0. They come up UpToDate with a fresh UUID and nothing is resynced.When the peers connect, the kernel logs
uuid_compare()=unrelated-data by rule=history-bothandUnrelated data, aborting!, and the connections go StandAlone. After that piraeus-ha-controller logsresource '<pv>' on node '<node>' has failed, evictingevery minute and keeps deleting the VolumeAttachment. The node also gets thedrbd.linbit.com/lost-quorum:NoScheduletaint, so nothing without a toleration is scheduled there until the volume is deleted.In one run the reader pod got an empty replica:
The snapshot itself is fine. I cloned the ZFS snapshot by hand on every node and read it with
drbdmeta ... internal dump-md: the metadata is valid andcurrent-uuidis the same on all three.Additional context
linstor-csi restores on one node on purpose and then autoplaces the rest. See
reconcileSnapshotResourcesinpkg/client/linstor.goat v1.11.2, which callsRestoreSnapshotwithNodes: []string{selectedNode}, followed byreconcileResourcePlacement.LINSTOR creates the autoplaced replicas as brand new volumes. In the satellite's
DrbdLayer.createMetaData(v1.33.3) a new replica either wins the initial UpToDate race and gets a random current UUID, or asksDrbdLayerUtils.skipInitSync. That one returns true for any ZFS or ZFS_THIN volume unlessDrbdOptions/ForceInitialSyncPermanentis set. For a resource definition that already holds data this shortcut is wrong, the new replica has to sync from the restored one.A possible mitigation, not tested yet: set
DrbdOptions/ForceInitialSync=trueon the resource group. The controller copies it intoForceInitialSyncPermanentwhen it creates a resource (CtrlRscCrtApiHelper.copyForceInitialSyncProp), and thenskipInitSyncreturns false. The price is a full initial sync for every new ZFS volume, so it needs measuring before it goes into platform defaults. Restoring on every node that has the snapshot would avoid the empty replicas too.I'm preparing an upstream report for LINSTOR and will link it here.