Skip to content

Commit 238a3b6

Browse files
kvapsclaude
kvaps
and
claude
committed
feat(kubernetes): default NVreg_NvLinkDisable=1 for tenant gpu-operator
When a single SXM GPU (e.g. H200) is passed through into a tenant worker VM without the NVSwitches, the nvidia driver waits forever for an NVLink fabric that can never come up (Fabric State: In Progress). Every CUDA call then fails with 'system not yet initialized' and the nvidia-operator-validator crashloops. Loading the kernel module with NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes — correct for per-GPU passthrough, which has no in-VM NVLink/NVSwitch peering anyway. The upstream gpu-operator chart only references an existing ConfigMap by name via driver.kernelModuleConfig.name and never creates it. Render that ConfigMap from the gpu-operator wrapper chart (opt-in via kernelModuleConfig.create) so it lands in the tenant cozy-gpu-operator namespace alongside the operator, and wire the tenant HelmRelease defaults to enable it and point the driver at it. Remains overridable via addons.gpuOperator.valuesOverride. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]>
1 parent 9c0e76d commit 238a3b6

4 files changed

Lines changed: 81 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,28 @@
11
{{- define "cozystack.defaultGpuOperatorValues" -}}
2-
{{- if .Values.addons.hami.enabled }}
2+
{{- /*
3+
GPU passthrough default: cozystack passes individual SXM GPUs (e.g.
4+
H200) into tenant worker VMs WITHOUT the NVSwitches. The nvidia driver
5+
then waits forever for an NVLink fabric that can never come up (no
6+
NVSwitch / fabricmanager), leaving Fabric State: In Progress — every
7+
CUDA call fails with "system not yet initialized" and the
8+
nvidia-operator-validator crashloops. Loading the kernel module with
9+
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes.
10+
This is correct for per-GPU passthrough since there is no in-VM
11+
NVLink/NVSwitch peering anyway.
12+
13+
The upstream chart only references an existing ConfigMap by name; the
14+
wrapper chart (packages/system/gpu-operator) renders that ConfigMap
15+
when kernelModuleConfig.create is true. Overridable via
16+
addons.gpuOperator.valuesOverride (mergeOverwrite below keeps user
17+
values winning).
18+
*/}}
19+
kernelModuleConfig:
20+
create: true
321
gpu-operator:
22+
driver:
23+
kernelModuleConfig:
24+
name: nvidia-kernel-module-params
25+
{{- if .Values.addons.hami.enabled }}
426
devicePlugin:
527
enabled: false
628
{{- end }}

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ tests:
1818
path: spec.values.gpu-operator.devicePlugin.enabled
1919
value: false
2020

21-
- it: should not have values when hami is disabled and no overrides
21+
- it: should default the NvLinkDisable kernel module config even when hami is disabled
2222
set:
2323
addons:
2424
gpuOperator:
@@ -28,8 +28,35 @@ tests:
2828
enabled: false
2929
valuesOverride: {}
3030
asserts:
31+
- equal:
32+
path: spec.values.kernelModuleConfig.create
33+
value: true
34+
- equal:
35+
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
36+
value: nvidia-kernel-module-params
3137
- notExists:
32-
path: spec.values
38+
path: spec.values.gpu-operator.devicePlugin
39+
40+
- it: should let user override the kernel module config name
41+
set:
42+
addons:
43+
gpuOperator:
44+
enabled: true
45+
valuesOverride:
46+
gpu-operator:
47+
driver:
48+
kernelModuleConfig:
49+
name: my-custom-kmc
50+
hami:
51+
enabled: false
52+
valuesOverride: {}
53+
asserts:
54+
- equal:
55+
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
56+
value: my-custom-kmc
57+
- equal:
58+
path: spec.values.kernelModuleConfig.create
59+
value: true
3360

3461
- it: should apply hami defaults when valuesOverride key is omitted
3562
set:

‎packages/system/gpu-operator/templates/kernel-module-config.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{{- if .Values.kernelModuleConfig.create }}
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{ .Values.kernelModuleConfig.name }}
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
app.kubernetes.io/managed-by: {{ .Release.Service }}
9+
app.kubernetes.io/name: gpu-operator
10+
data:
11+
nvidia.conf: |
12+
{{- .Values.kernelModuleConfig.content | nindent 4 }}
13+
{{- end }}

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,19 @@ gpu-operator:
2525
# values-vgpu.yaml.
2626
vgpuDeviceManager:
2727
enabled: false
28+
29+
# Optional ConfigMap holding extra nvidia kernel module parameters.
30+
# The upstream chart only references an existing ConfigMap by name via
31+
# driver.kernelModuleConfig.name — it never creates it. When create is
32+
# true this wrapper chart renders the ConfigMap into the release
33+
# namespace so the params actually land. The driver entrypoint reads
34+
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
35+
# options. Used by tenant Kubernetes clusters to set
36+
# NVreg_NvLinkDisable=1 (see packages/apps/kubernetes gpu-operator
37+
# HelmRelease defaults); off by default for host-level passthrough/vgpu
38+
# variants where the in-host driver is disabled.
39+
kernelModuleConfig:
40+
create: false
41+
name: nvidia-kernel-module-params
42+
content: |
43+
NVreg_NvLinkDisable=1

0 commit comments

Comments
 (0)