feat(kubernetes): default gpu-operator to an OS-provided driver on Talos - #3585

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

Merge Risk: ⚪ Minimal · up to 669fb

This change sets Talos GPU Operator defaults to use host-provided NVIDIA drivers and CDI. No blocking issue was found, and the author reports a successful hardware test.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 669fb

The change avoids competing driver installations, but GPU availability now depends on the host driver and CDI configuration being ready together. The available evidence does not establish a new privilege exposure or verify recovery from a partial rollout.

Retained concerns

• Low · reliability · inferred: The new defaults remove operator ownership of driver and toolkit setup, but the HelmRelease does not explicitly gate deployment on host driver and CDI readiness. A partial transition could leave GPU workloads without the expected library injection; cleanup and rollback to a single driver owner remain unverified.
• Low · reliability · observed: An existing partial devicePlugin.env override replaces all new CDI defaults rather than extending them. Such an override can silently select a different injection strategy during rollout; the test confirms this behavior but does not guard the CDI contract.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

End-to-end run done, on hardware, and it holds. Reporting it as promised rather than leaving the claim on unit tests alone.

Setup: a fresh single-node tenant cluster, one node group with an L40S (nvidia.com/AD102GL_L40S), the open-kernel-modules schematic, machine.kernel.modules added by hand since #3571 is not released yet, and no DaemonSet of any kind. The values under test were applied through addons.gpuOperator.valuesOverride with content identical to this PR's default.

The last row matters because a Talos node whose extension services cannot start reboots at roughly 70 minutes of uptime while still reporting Ready; 93 minutes on a single boot rules that out. Note a one-time reboot right after the install is normal and is not that failure — the node object is younger than its CAPI Machine, which is expected.

The L40S is the point: everything behind this configuration was previously only exercised on Blackwell (GB202). It works unchanged on Ada, so the set is not GPU-generation-specific.

What this run does not establish, stated plainly:

• It exercised the values, not this PR's default. The cluster runs the released chart, so the set arrived through valuesOverride; that the chart emits it by default is covered by the unit tests here and nothing more. Testing the default itself needs a release carrying this PR.
• It says nothing about whether NVreg_NvLinkDisable=1 is needed. An L40S is PCIe and has no NVLink, so the run shows the parameter is harmless and that CUDA initialises with it. Its necessity rests on the SXM passthrough case in production, not on this cluster.
• CDI injection was confirmed by its effect, not by inspecting containerd. Worth recording for whoever debugs this next: with DEVICE_LIST_STRATEGY=cdi-annotations no CDI annotation appears on the Pod object — the device plugin returns it in the Allocate response and kubelet passes it to the CRI runtime. Its absence from kubectl get pod -o yaml is expected and is not evidence that CDI failed. The evidence is nvidia-smi working inside a pod whose image ships no driver.
• HAMi remains untested and uncovered, as the PR body says.

Taking this out of draft on the strength of the above.

Two findings from the review: one taken, one I disagree with.

Taken — the missing assertions. 48007bd pins NVIDIA_DRIVER_ROOT, DEVICE_ID_STRATEGY: uuid and CDI_ANNOTATION_PREFIX: cdi.k8s.io/ on the device plugin, plus the validator's own NVIDIA_DRIVER_ROOT alongside DRIVER_ROOT. The reasoning is right: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of the three breaks device resolution while the two original assertions still pass.

Not taken — emptying kernelModuleConfig.content. I think this would make things worse rather than remove staleness.

That content is only read when an operator sets both driver.enabled: true and kernelModuleConfig.create: true, which is to say when they have deliberately re-enabled the driver container. In that configuration NVreg_NvLinkDisable=1 is the correct value for Cozystack, not an accident: individual GPUs are passed into worker VMs without the NVSwitches, so the driver otherwise waits forever for an NVLink fabric that can never come up — Fabric State stays In Progress and every CUDA call fails with system not yet initialized. Emptying the default means an operator who re-enables the driver container silently loses the parameter and lands in exactly that failure, which is the class of silent breakage this PR and #3571 exist to remove.

There is also no contradiction to resolve. "No variant enables it any more" and "the content is right when it is enabled" are both true at once: create: false already makes it inert, and content is the whole payload, so an operator who wants different parameters overrides that key and replaces it wholesale rather than merging into it. Nothing is silently inherited.

The comment above it was updated in this PR to say precisely that nothing enables it by default and that it is kept for an operator who re-enables the driver container, who then also has to set create: true. If that reads as stale rather than as a documented escape hatch, the fix is in the wording, and I would rather sharpen the comment than remove a default that is correct whenever it is reachable.

Aleksei Sviridkin (@lexfrei) all three blockers are fixed in 669fb24, pushed as a single commit.

B1. Every nodeGroups.<name>.kernelModules reference is gone: from the template comment, the test comment, packages/system/gpu-operator/values.yaml, the "removed default" section of the PR body and the release note. They now name the top-level kernelModules field of the KubernetesNodes app in packages/apps/kubernetes-nodes (#3571). The body and the release note also say that the field's automatic set for a pool holding an nvidia.com/* GPU already loads nvidia with NVreg_NvLinkDisable=1, since that is the answer to "where did the parameter go".

B2. I squashed 61dce5d into d105bbe. The CDI assertions it added are now justified in one paragraph of the single commit body, with no review wording. hack/check-commit-trailers.sh passes.

B3. The template comment is down from about 50 lines to 27. It keeps only the constraints: why driver and toolkit are off, why the validator is pointed at the host root, why CDI is the only route (and the pod-admission timing), and the list-replace trap in mergeOverwrite. The history paragraph and the production-validation claim are gone, and so is the NOTE:. In the test file, "gap 3" and "gap 4" and the "whole contract" and "as load-bearing as the two above" lines are replaced by the reason each assertion exists. The ConfigMap test is renamed from "no longer renders" to "does not render". In the system values, "No variant enables it any more… used to" now just says how tenant clusters use the chart.

Nothing changed in behaviour. The rendered HelmRelease is byte-identical to 61dce5d with HAMi on and with HAMi off, and helm unittest passes: 151 tests in packages/apps/kubernetes, 3 in packages/system/gpu-operator.