Skip to content

kubevirt-csi node: NFS publish and unpublish can still block on a dead server #4814

Description

The kubevirt-csi node plugin can still get stuck on a dead NFS server in a few places after the soft temp mount fix. That fix is #4791. It mounts the temp NFS root soft and allows one publish per target path at a time. The guard keeps each case below to one stuck call per target, but the call never ends.

  • A publish whose target path is already hard-mounted from the dead server blocks for good in the IsLikelyNotMountPoint check at the start of NodePublishVolume.
  • Unpublish goes to the upstream NodeService method, which does not look at the guard. It can run against a target that a stuck publish still holds.
  • Over NFSv4 a soft RPC fails only once the TCP connection is declared dead. If the connection stays up while the NFS service hangs, the temp mount's RPCs do not time out.
  • The Mount call from mount-utils runs mount with no deadline.
  • After the unmount budget runs out, umount.nfs can stay stuck in umount2(), and a stat may run before umount2. Both come from the follow-up comment on kubevirt-csi node: NFS publish hangs on the temp mount when the NFS server stops answering #4581. The soft temp mount may end them sooner, but I have not verified that.
  • Moving old files into /data checks the destination and then renames. An entry that appears in /data between the two is replaced by the rename. As far as I know the NFS client has no RENAME_NOREPLACE, so there is no simple fix.

All of it is in packages/apps/kubernetes/images/kubevirt-csi-driver/node.go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions