Describe the bug
Restarting the kubevirt-csi node plugin in a tenant cluster breaks every RWX volume that is mounted on that node, for good. Any upgrade that changes the plugin image restarts it, so every such upgrade does this. The pods keep running but their NFS mount answers Network is unreachable, and a pod using it cannot be deleted afterwards.
Environment
- Cozystack main at
1d813c944 with the kubevirt-csi-driver image rebuilt from an open dependency roll-up. Only the image changed, the DaemonSet template is the one from main.
- Test cluster, tenant Kubernetes v1.35 on Talos, one worker.
- RWX PVC on the tenant
replicated StorageClass, served by linstor-csi-nfs.cozy-linstor.svc.
To Reproduce
- In a tenant cluster, create an RWX PVC and a pod that mounts it, and write a file.
- Restart the node plugin pod, for example with
kubectl -n cozy-csi rollout restart ds/kubevirt-csi-node. I hit it by changing the image.
- Read the file from the pod again, then delete the pod.
Expected behaviour
The mount keeps working after the plugin restarts. Deleting the pod unmounts the volume.
Actual behaviour
About 100 seconds after the plugin pod was replaced, the tenant node's kernel logged:
NFS: state manager: check lease failed on NFSv4 server linstor-csi-nfs.cozy-linstor.svc with error 101
Error 101 is ENETUNREACH. From then on every read from the pod fails:
sha256sum: can't open '/rwx/blob': Network is unreachable
ext4 and xfs volumes on the same node were not affected.
Deleting the pod never finishes. NodeUnpublishVolume fails on every retry:
failed to cleanup mount point Error checking path: stat /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount: network is unreachable
Additional context
The plugin mounts NFS from inside its own container (NodePublishVolume in packages/apps/kubernetes/images/kubevirt-csi-driver/node.go), and the DaemonSet in packages/system/kubevirt-csi-node/templates/deploy.yaml does not set hostNetwork. The kernel NFS client is tied to the network namespace of the process that mounted it. The mount showed clientaddr= equal to the plugin pod's IP. When the pod is replaced, its network namespace is gone, and with the fatal_neterrors=ENETDOWN:ENETUNREACH mount option the error is final.
The unpublish hang is a second problem on top of that. mount.CleanupMountPoint only unmounts a path whose stat error passes IsCorruptedMnt. In k8s.io/mount-utils v0.37.1 that list has ENOTCONN, ESTALE, EIO, EACCES, EHOSTDOWN, EWOULDBLOCK and ENODEV, but not ENETUNREACH. So the stat error is returned as is and the mount stays forever.
Two changes look needed. The NFS mount has to live in the node's network namespace, either with hostNetwork: true on the node plugin (it then needs a working DNS policy for the .svc name) or by mounting through a helper that runs in the host namespace. Unpublish should also treat ENETUNREACH as a dead mount and unmount it (lazy or forced). #4581 and #4814 are about a dead NFS server, which is a different trigger for similar hangs.
Describe the bug
Restarting the kubevirt-csi node plugin in a tenant cluster breaks every RWX volume that is mounted on that node, for good. Any upgrade that changes the plugin image restarts it, so every such upgrade does this. The pods keep running but their NFS mount answers
Network is unreachable, and a pod using it cannot be deleted afterwards.Environment
1d813c944with the kubevirt-csi-driver image rebuilt from an open dependency roll-up. Only the image changed, the DaemonSet template is the one from main.replicatedStorageClass, served bylinstor-csi-nfs.cozy-linstor.svc.To Reproduce
kubectl -n cozy-csi rollout restart ds/kubevirt-csi-node. I hit it by changing the image.Expected behaviour
The mount keeps working after the plugin restarts. Deleting the pod unmounts the volume.
Actual behaviour
About 100 seconds after the plugin pod was replaced, the tenant node's kernel logged:
Error 101 is ENETUNREACH. From then on every read from the pod fails:
ext4 and xfs volumes on the same node were not affected.
Deleting the pod never finishes. NodeUnpublishVolume fails on every retry:
Additional context
The plugin mounts NFS from inside its own container (
NodePublishVolumeinpackages/apps/kubernetes/images/kubevirt-csi-driver/node.go), and the DaemonSet inpackages/system/kubevirt-csi-node/templates/deploy.yamldoes not sethostNetwork. The kernel NFS client is tied to the network namespace of the process that mounted it. The mount showedclientaddr=equal to the plugin pod's IP. When the pod is replaced, its network namespace is gone, and with thefatal_neterrors=ENETDOWN:ENETUNREACHmount option the error is final.The unpublish hang is a second problem on top of that.
mount.CleanupMountPointonly unmounts a path whosestaterror passesIsCorruptedMnt. In k8s.io/mount-utils v0.37.1 that list has ENOTCONN, ESTALE, EIO, EACCES, EHOSTDOWN, EWOULDBLOCK and ENODEV, but not ENETUNREACH. So the stat error is returned as is and the mount stays forever.Two changes look needed. The NFS mount has to live in the node's network namespace, either with
hostNetwork: trueon the node plugin (it then needs a working DNS policy for the.svcname) or by mounting through a helper that runs in the host namespace. Unpublish should also treat ENETUNREACH as a dead mount and unmount it (lazy or forced). #4581 and #4814 are about a dead NFS server, which is a different trigger for similar hangs.