Repository navigation
kubevirt-csi node: NFS publish hangs on the temp mount when the NFS server stops answering #4581
Description
The kubevirt-csi node plugin can hang an NFS volume publish if the NFS server stops answering while the plugin works on its temporary mount.
To create /data, NodePublishVolume in packages/apps/kubernetes/images/kubevirt-csi-driver/node.go mounts the export root into an nfs-init-* temp dir. The mount options are only nfsvers=4.2 and port=, so it is a hard mount. After that the plugin calls os.MkdirAll, os.ReadDir and os.Rename on it. None of these calls has a deadline on a hard mount, and neither does the Mount before them. If the server goes away at that point, the handler goroutine blocks.
Kubelet gives up after its 2m CSI deadline (csiTimeout in pkg/volume/csi/csi_plugin.go) and retries the call. The blocked goroutine stays. Each retry creates another temp dir and blocks the same way. Stuck goroutines and nfs-init-* dirs pile up until the server comes back or the container restarts.
The PR for #4561 bounds the unmount at the end of this path. It does not help with the calls before it. I see two ways to fix this: mount the temp root with soft and a bounded timeo/retrans, since it only does metadata work, or refuse a new publish for a volume that already has one in progress.
Activity
lexfrei commentedon Oct 5, 2026
Two more things in the same area are left after #4584. Both are bounded by its 30s unmount budget, so neither hangs the publish, but both pile up per retry the same way as the calls above.
First, when the plain
umounthits its timeout, onlyumountgets SIGKILL. Theumount.nfshelper it forked stays behind, stuck inumount2()on the dead server, and the plugin goroutine waiting on the unmount stays too, one of each per publish that timed out. mount-utils'UnmountWithForceleaves the same orphan. I saw the orphan with a fakeumountthat forks a child, not on a real dead server.Second, I have not verified this one. util-linux
umount, or theumount.nfshelper, maystatthe target before it callsumount2(MNT_FORCE). On a hard mount with the server gone thatstatblocks, soumount -fwould hang before it does anything and only the timeout would end it. To check: on almalinux:9 with nfs-utils, mount an export withoutsoft, drop the server (for example with an iptables DROP rule on port 2049), runstrace -f umount -f <dir>and see whether it reachesumount2or blocks instat/statxfirst. If it blocks there,umount -c(no canonicalize) may get past it.The
softmount with a boundedtimeo/retranssuggested above would likely cover both, since RPCs on the temp mount would then fail instead of blocking.fix(kubernetes): force the kubevirt-csi temp NFS unmount after a time…
fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)