kubevirt-csi-driver: stale DataVolume in VM spec after rate-limited ControllerPublishVolume causes permanent attach failure #2634

Summary

When ControllerPublishVolume fails due to a rate limiter timeout after AddVolume has already been called on the KubeVirt API, the DataVolume is written into VM.spec.template.spec.volumes but never becomes Ready in VMI volumeStatus. Subsequent ControllerUnpublishVolume calls return success without removing the volume from VM spec. When the same volume is later attached to a different VM, the anti-split-brain check blocks all further attachments permanently.

Root cause

ControllerPublishVolume has two phases:

1. Call AddVolume(persist=true) — writes DataVolume to VM.spec.template.spec.volumes
2. Wait up to 2 minutes for the volume to appear in VMI.status.volumeStatus as Ready

If the rate limiter fires during phase 2, the RPC returns an error. The DataVolume is already in the VM spec, but external-attacher receives a failure.

ControllerUnpublishVolume checks VMI.status.volumeStatus to decide whether to call RemoveVolume. If the volume is absent or in a non-Ready state (e.g. Bound), the implementation returns success immediately without calling RemoveVolume. The stale entry remains in VM.spec.template.spec.volumes.

When a subsequent ControllerPublishVolume attaches the same volume to a different VM, both VMs have the DataVolume in spec simultaneously. The anti-split-brain check then blocks all future attachments:

ControllerPublishVolume failed: RWX block volume <namespace>/<pvc> is being used
by pods from different VMs (<vm-a> and <vm-b>); this is not supported — RWX block
volumes with allow-two-primaries are only for live migration of a single VM

The hp-volume pods stay in ContainerCreating, virt-handler never issues device_add to QEMU, and the tenant cluster kubelet sees:

MountVolume.MountDevice failed: rpc error: code = Unknown
desc = couldn't find device by serial id

Conditions that trigger this

• Tenant cluster scale-out: multiple new worker VMs joining simultaneously
• Multiple PVCs with storageClass: kubevirt attaching at the same time
• CSI rate limiter is hit during the burst

Expected behavior

ControllerUnpublishVolume must remove the DataVolume from VM.spec.template.spec.volumes regardless of the volume's state in VMI.status.volumeStatus. The VM spec is the authoritative record for persistent hotplug volumes and must be cleaned up independently of VMI status.

Actual behavior

ControllerUnpublishVolume returns success without calling RemoveVolume if the volume is not in Ready state in VMI volumeStatus. The VM spec retains the stale DataVolume entry permanently.

Manual remediation

virtctl removevolume <vm-name> \
  --volume-name=<datavolume-name> \
  --namespace <namespace>

Run for each stale DataVolume on the old VM.

Environment

• Cozystack: v1.1.3
• KubeVirt: v1.6.3
• kubevirt-csi-driver: ghcr.io/cozystack/cozystack/kubevirt-csi-driver:0.0.0
• csi-attacher sidecar: quay.io/openshift/origin-csi-external-attacher:latest