Skip to content

Commit 7c8a254

Browse files
committed
feat(gpu-operator): add container variant for preinstalled host driver
The two existing variants assume the host has neither the NVIDIA driver nor the container toolkit: default unbinds the host driver to bind vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host driver and a license server. Neither path produces a working setup on a node that already ships the driver and nvidia-container-toolkit via the distro package manager — the most common shape of a Linux GPU node. The new container variant surfaces the third architectural mode that upstream NVIDIA gpu-operator already supports: sandboxWorkloads disabled, devicePlugin enabled, with every component that would conflict with the host install switched off. - driver.enabled=false use the host driver - toolkit.enabled=false host already has nvidia-container-toolkit; enabling overwrites containerd config.toml and the CDI hooks the host package wired - devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet - vfioManager.enabled=false do not unbind the host driver - sandboxWorkloads.enabled=false regular pod workloads, not VMs Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent d3f282c commit 7c8a254

6 files changed

Lines changed: 110 additions & 30 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,11 @@ This guide focuses on the **SR-IOV path**, which is the only model NVIDIA suppor
2323

2424
## Variants
2525

26-
The `gpu-operator` package exposes two variants:
26+
The `gpu-operator` package exposes three variants. This document is vGPU-focused; the variant inventory is shared.
2727

2828
- **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host.
2929
- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
30+
- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs.
3031

3132
## Building the vGPU Manager image
3233

@@ -231,10 +232,12 @@ Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Doc
231232

232233
## OS support summary
233234

234-
| Host OS | passthrough (`default`) | vGPU (`vgpu`) |
235-
| --- | --- | --- |
236-
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) |
237-
| Ubuntu 22.04 | ✅ | ✅ |
238-
| Ubuntu 20.04 | ✅ | ✅ |
239-
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork |
240-
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked |
235+
The `container` variant column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager. On Talos the operator-validator's host-driver auto-detection probes `/host/usr/bin/nvidia-smi`, which does not exist on Talos (the extension installs into `/usr/local/bin/`) — see `packages/system/gpu-operator/examples/` for the Talos-specific path with a compat DaemonSet and `hostPaths.driverInstallDir` override.
236+
237+
| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) |
238+
| --- | --- | --- | --- |
239+
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit |
240+
| Ubuntu 22.04 | ✅ | ✅ | ✅ |
241+
| Ubuntu 20.04 | ✅ | ✅ | ✅ |
242+
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork | ✅ |
243+
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host-driver auto-detect path differs — use `examples/values-native-talos.yaml` as a starting point instead |

‎hack/check-gpu-operator-variants.bats‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,32 @@
11
#!/usr/bin/env bats
22
# -----------------------------------------------------------------------------
3-
# Catches silent regressions in the gpu-operator package's two variants:
3+
# Catches silent regressions in the gpu-operator package's three variants:
44
#
55
# 1. The base values.yaml pins ccManager.enabled=false and
66
# vgpuDeviceManager.enabled=false. Both upstream defaults are now true
77
# (chart v26.x). If the next chart bump silently un-pins either one,
88
# the `default` (passthrough) variant would gain confidential-computing
99
# auto-enable on Hopper hardware (ccManager) or render an mdev DaemonSet
10-
# that crashloops on Ada+/Blackwell (vgpuDeviceManager).
10+
# that crashloops on Ada+/Blackwell (vgpuDeviceManager). The same union
11+
# must hold for the `container` variant; the `vgpu` variant re-affirms
12+
# vgpuDeviceManager=false in its own overlay for the same reason.
1113
#
12-
# 2. Each variant's overlay must set sandboxWorkloads.defaultWorkload to its
13-
# own enum value (vm-passthrough / vm-vgpu) — without it, GPU nodes that
14-
# lack the per-node label nvidia.com/gpu.workload.config fall back to
15-
# the upstream default 'container' workload, no DaemonSet renders, and
16-
# the variant is silently a no-op.
14+
# 2. The sandbox variants must set sandboxWorkloads.defaultWorkload to
15+
# their own enum value (vm-passthrough / vm-vgpu) — without it, GPU
16+
# nodes that lack the per-node label nvidia.com/gpu.workload.config
17+
# fall back to the upstream default 'container' workload, no DaemonSet
18+
# renders, and the variant is silently a no-op. The container variant
19+
# relies on the same upstream default but in the opposite direction:
20+
# with sandboxWorkloads.enabled=false the chart picks defaultWorkload
21+
# 'container', which is what that variant wants — pinned here so an
22+
# upstream rename does not silently break it.
1723
#
1824
# 3. The vgpu variant must enable the vGPU manager DaemonSet
19-
# (vgpuManager.enabled=true) and the passthrough variant must keep
20-
# driver.enabled / devicePlugin.enabled at false.
25+
# (vgpuManager.enabled=true), the passthrough variant must keep
26+
# driver.enabled / devicePlugin.enabled at false, and the container
27+
# variant must keep driver / toolkit / vfioManager at false (host
28+
# already provides them) while keeping devicePlugin enabled (publishes
29+
# nvidia.com/gpu to the kubelet).
2130
#
2231
# Implementation: render the chart with the merged values for each variant
2332
# and grep for the relevant ClusterPolicy keys. We strip the package's
@@ -96,3 +105,26 @@ render_variant() {
96105
grep -A 1 '^ driver:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
97106
grep -A 1 '^ devicePlugin:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
98107
}
108+
109+
@test "container variant: ccManager and vgpuDeviceManager pinned off" {
110+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
111+
render_variant container
112+
grep -A 1 '^ ccManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
113+
grep -A 1 '^ vgpuDeviceManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
114+
}
115+
116+
@test "container variant: sandboxWorkloads off, defaultWorkload stays upstream 'container'" {
117+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
118+
render_variant container
119+
grep -A 2 '^ sandboxWorkloads:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
120+
grep -A 2 '^ sandboxWorkloads:' "$TMP/rendered.yaml" | grep -q 'defaultWorkload: container'
121+
}
122+
123+
@test "container variant: driver, toolkit, vfioManager off; devicePlugin on" {
124+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
125+
render_variant container
126+
grep -A 1 '^ driver:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
127+
grep -A 1 '^ toolkit:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
128+
grep -A 1 '^ vfioManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
129+
grep -A 1 '^ devicePlugin:' "$TMP/rendered.yaml" | grep -q 'enabled: true'
130+
}

‎packages/core/platform/sources/gpu-operator.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,16 @@ spec:
3636
privileged: true
3737
namespace: cozy-gpu-operator
3838
releaseName: gpu-operator
39+
- name: container
40+
dependsOn:
41+
- cozystack.networking
42+
components:
43+
- name: gpu-operator
44+
path: system/gpu-operator
45+
valuesFiles:
46+
- values.yaml
47+
- values-container.yaml
48+
install:
49+
privileged: true
50+
namespace: cozy-gpu-operator
51+
releaseName: gpu-operator

‎packages/system/gpu-operator/examples/README.md‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,10 @@
11
# GPU operator — native pod workload on Talos (reference)
22

3-
The files in this directory are **not** templates. They are reference
4-
artifacts that document one working configuration for running GPU
5-
workloads directly in pods on a Talos-based Cozystack cluster, together
6-
with the DCGM metrics needed by the `gpu/gpu-performance` Grafana
7-
dashboard.
8-
9-
The out-of-the-box `values-passthrough.yaml` for this package targets the
10-
sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here
11-
illustrate an alternative — running CUDA workloads in regular pods with
12-
the NVIDIA device plugin — and the workarounds it currently requires on
13-
Talos.
3+
The files in this directory are **not** templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the `gpu/gpu-performance` Grafana dashboard.
4+
5+
For non-Talos Linux hosts that already provide the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class `container` variant instead — `variant: container` in your Package CR is enough and no `hostPaths.driverInstallDir` override is needed because the operator-validator auto-detects the host driver at `/host/usr/bin/nvidia-smi`. The files here remain the right starting point on Talos, where the extension installs `nvidia-smi` at `/usr/local/bin/` and the auto-detect path misses it.
6+
7+
The out-of-the-box `values-passthrough.yaml` for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.
148

159
## Files
1610

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Container variant: GPU workloads run in regular pods on hosts that
2+
# already have the NVIDIA driver and nvidia-container-toolkit installed
3+
# (typical Ubuntu/Debian shape: apt-installed driver + apt-installed
4+
# nvidia-container-toolkit). The third architectural mode that upstream
5+
# NVIDIA gpu-operator supports — sandboxWorkloads disabled,
6+
# devicePlugin enabled — surfaced here as a Cozystack variant.
7+
gpu-operator:
8+
sandboxWorkloads:
9+
# Default upstream workload on a node without the per-node
10+
# 'nvidia.com/gpu.workload.config' label is 'container' when
11+
# sandboxWorkloads is off — which is exactly what this variant
12+
# wants. No defaultWorkload override needed; the counterpart
13+
# overrides in values-passthrough.yaml / values-vgpu.yaml exist
14+
# only because those variants flip sandboxWorkloads to true.
15+
enabled: false
16+
driver:
17+
# Host already runs the NVIDIA driver (apt-installed). The
18+
# operator's driver DaemonSet would attempt to load its own
19+
# kernel module and clash with the host one.
20+
enabled: false
21+
devicePlugin:
22+
# Publish nvidia.com/gpu to the kubelet so pods can request GPUs.
23+
enabled: true
24+
vfioManager:
25+
# Defensive: vfioManager unbinds the host driver to bind vfio-pci.
26+
# With the host driver present and devicePlugin enabled, vfio
27+
# binding must not happen. Upstream defaults this to off when
28+
# sandboxWorkloads is disabled, but the explicit pin protects
29+
# against an upstream default flip.
30+
enabled: false
31+
toolkit:
32+
# Host already has nvidia-container-toolkit installed (the typical
33+
# apt-installed shape). Running the operator's toolkit DaemonSet
34+
# would overwrite /etc/containerd/config.toml and the CDI hooks
35+
# already wired by the host package, breaking the host install.
36+
# NVIDIA's documentation recommends disabling this when the
37+
# toolkit is preinstalled on the node.
38+
enabled: false

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Defaults applied to every variant of the gpu-operator package.
2-
# The variant-specific files (values-passthrough.yaml, values-vgpu.yaml)
3-
# layer on top.
2+
# The variant-specific files (values-passthrough.yaml, values-vgpu.yaml,
3+
# values-container.yaml) layer on top.
44
gpu-operator:
55
# Upstream chart v26.x flips ccManager.enabled to true and
66
# ccManager.defaultMode to "on". On Hopper-class hardware

0 commit comments

Comments
 (0)