bug(gpu-operator): passthrough variant silently skips VFIO bind when host has a pre-installed NVIDIA driver #2763

Summary

When cozystack.gpu-operator is enabled with the default (passthrough) variant on a node that already has the NVIDIA host driver installed (the common state for a Linux host provisioned via apt install nvidia-driver-* / nvidia-driver-*-open), the vfio-manager DaemonSet's k8s-driver-manager init container detects the host driver, labels the node nvidia.com/gpu.deploy.driver=pre-installed, exits success — and skips the unbind/bind step entirely. nvidia-sandbox-validator then crashloops because the VFIO bind never happened.

There is no warning at install time, no opt-in flag to force the rebind, and no documentation that the passthrough variant assumes a clean host.

Reproduction

1. Provision an Ubuntu 24.04 host with apt install nvidia-driver-580-open (or any host driver), confirm nvidia-smi works.
2. Enable cozystack.gpu-operator in bundles.enabledPackages of the Platform Package. Default variant resolves to default (passthrough).
3. Wait for the HelmRelease to install.

Observed

# vfio-manager k8s-driver-manager init log:
Host driver detected: 580.82.07
NVIDIA GPU driver is already pre-installed on the node,
  disabling the containerized driver
Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed

vfio-manager exits success without doing anything. nvidia-sandbox-validator then loops with:

Error: error validating vfio-pci driver installation:
  device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia'

lspci -nnk -d 10de:2335 still shows Kernel driver in use: nvidia. No GPU is available for KubeVirt VM passthrough.

Expected

Either (a) the default variant should refuse to install with a clear error pointing at the host-driver-present condition, or (b) there should be a values knob (e.g. forceUnbindHostDriver: true) that tells k8s-driver-manager to take ownership of the GPU anyway, or (c) the documentation for values-passthrough.yaml should state that the variant requires a clean host.

The current silent-skip + downstream crashloop is the worst of all worlds: no signal at the point of decision, and a confusing failure mode several layers down.

Workaround

Remove the host NVIDIA stack before enabling the passthrough variant: apt purge the entire nvidia-driver-* + libnvidia-* + CUDA toolkit set, blacklist nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm,nouveau in /etc/modprobe.d/, reboot. Use an explicit package list — autoremove: true can sweep NVIDIA DOCA / Mellanox / InfiniBand userspace out at the same time because of the shared vendor prefix.

After reboot, delete the stuck nvidia-vfio-manager-* and nvidia-sandbox-* pods so the DaemonSets re-create them against the cleaned host. Within a couple of minutes vfio-manager binds all GPUs to vfio-pci and node Allocatable shows the registered resource.

Related

• A container variant of cozystack.gpu-operator would give pre-installed-driver hosts a working path without needing to purge the driver (separate issue).
• Wiring the registered GPU resource into KubeVirt's permittedHostDevices is also currently manual (separate issue).