Repository navigation
GPU node groups on Talos workers: kernel modules, NVIDIA schematic, gpu-operator wiring, CDI #3563
Description
Activity
Picking up gap 1 (the kernel-modules values surface). Before writing it I would like the API shape arbitrated here rather than decided in the PR, because
terraform-provider-cozystacktranscribesvalues.schema.jsonby hand and a later rename is a breaking change on that side too.Three shapes, all of which emit
machine.kernel.modulesinto the worker machine config thatpackages/apps/kubernetesandpackages/apps/kubernetes-nodesboth apply:1. Explicit field only. A per-node-group
kernelModules, mirroring the Talos structure:nodeGroups: gpu: gpus: - name: nvidia.com/GB202_RTX_PRO_6000 kernelModules: - name: nvidia - name: nvidia_uvm - name: nvidia_drm - name: nvidia_modeset
Honest and provider-agnostic, and it is the only shape that also serves non-GPU uses. It does not fix "not usable as shipped", though: every GPU user still has to know those four names and their order, which is exactly the knowledge that is missing from the docs today.
2. Automatic emission only. When a node group has
gpuswhose resource names are undernvidia.com/, emit the NVIDIA set. Fixes the default, but guesses on the operator's behalf and covers no other vendor (nor a node group that wants the modules without a passthrough device).3. Both.
kernelModulesis the API; when it is unset and the group hasnvidia.com/*GPUs, the chart defaults to the NVIDIA set. SettingkernelModulesexplicitly (including to[]) overrides the default, so the escape hatch is total.I would implement 3: it makes a GPU node group work as shipped, keeps a single documented field as the API surface the provider mirrors, and leaves no case where the chart's guess cannot be overridden. The trade-off is that
kernelModules: []becomes meaningful (an explicit opt-out), which has to be documented.The NVIDIA set validated in production, in this order —
nvidia,nvidia_uvm,nvidia_drm,nvidia_modeset.Two points I will handle regardless of the answer:
- The change lands in both charts, as flagged in the issue.
packages/apps/kubernetes-nodes/tests/render-parity.shexcludes the talos-reconcile Job output (its line 19), so nothing currently guarantees the two charts emit the same machine config — I will close that gap in the same PR rather than add a field into an unchecked duplication. - Sequencing per Aleksei Sviridkin (@lexfrei) on fix(kubernetes): name TalosConfigTemplate by content hash #3523: until the hash-named
TalosConfigTemplatelands, addingkernelModulesto an existing node group changes nothing (TalosConfigTemplate is immutable but applied on every reconcile, so machine-config changes never propagate #3515 — the apply is rejected server-side while the HelmRelease staysReady). Only node groups created afterwards get it. I will say so explicitly in the release note rather than let it read as a fix for running clusters.
Andrei Kvapil (@kvaps) Aleksei Sviridkin (@lexfrei) — option 1, 2 or 3? I will start on 3 and switch if you prefer another; the diff between them is small and local.
- The change lands in both charts, as flagged in the issue.
- added a commit that references this issue
on Aug 6, 2026 A fourth gap turned up in production this morning, and it changes the shape of gap 1's fix, so I have folded it into the same PR (#3571) rather than filing separately — the two are not independently usable.
spec.talos.schematicIDis cluster-wide, so a mixed cluster cannot work. A schematic is a fixed set of system extensions baked into one image, and Talos refuses to finish the boot sequence when an extension service in it cannot start:[talos] task startAllServices (1/1): service "ext-nvidia-cdi-gen" to be "up", service "ext-nvidia-persistenced" to be "up" [talos] task startAllServices (1/1): failed: 2 errors occurred: [talos] phase startEverything (9/9): failed [talos] boot sequence: failed -> rebootBoth services need an NVIDIA card. Setting the NVIDIA schematic for the sake of the GPU node group therefore puts every other node group into a permanent reboot cycle — observed on all four nodes of a non-GPU
md0, each at almost exactly 4206 s of uptime, in staggered phases, while the GPU nodes were fine at 23 h. Every pod on a node died together (SandboxChanged,TaintManagerEviction), giving identical restart counts per node, about 26 over 30 hours.This is why it matters for gap 1: an operator who follows the advice in this issue — put the NVIDIA extensions in the schematic, declare
machine.kernel.moduleson the GPU group — breaks every non-GPU group in the cluster the moment they do it. Documenting the schematic without per-group scoping would send people directly into this.Nothing surfaces it. kubelet starts before the failing phase, so the node holds
Readyandkubectl get nodeslooks clean.kubectl top nodesshows up to 96% memory just before each reboot, which reads as an OOM — it is page cache, andmin_over_time(node_memory_MemAvailable_bytes[30h])never approaches the limit. The discriminant is regularity: four nodes rebooting within seconds of the same uptime is a timer, not saturation. The boot failure itself is only visible on the serial console, which is silent between boots.So #3571 now carries a second commit adding an optional per-node-group
schematicIDthat falls back totalos.schematicID. Both consumers resolve it through one helper — the boot disk image the DataVolume pulls and the installer image in theTalosConfigTemplate— since a mismatch would make an in-place upgrade swap a node's extension set. UnlikekernelModulesit is deliberately not derived fromgpus: a schematic ID is an opaque image-factory digest, so the chart cannot know which one carries the NVIDIA extensions or synthesise one.Diagnostic that identifies it in one command, worth having in the docs: compare
dataVolumeTemplates[].spec.source.http.urlon each node group'sKubevirtMachineTemplatewith theinstall.imageof itsTalosConfigTemplate. A group where the two schematics differ is booting one extension set and would upgrade into another.Note this is a distinct cause from the other ~70 minute worker reboot cycle, where
apidnever gets its certificate because port 50001 is absent from the control plane Service. Worth checking that port before concluding it is the schematic; here it was published correctly.Happy to split this into its own issue if you would rather keep the paper trail separate from this umbrella — say so and I will, though the fix would stay in the same PR.
- added 3 commits that reference this issue
on Aug 6, 2026 Gaps 3 and 4 are now addressed, in #3585 — separate from #3571 because it touches the addon default rather than the machine config.
It turned out smaller than this issue implies. Neither gap needs a DaemonSet, a new image or a new package: the whole thing is the GPU Operator addon default, and the values are the ones already running in production, applied there by hand through
addons.gpuOperator.valuesOverride.- Gap 3 — the validator does have a mode for this after all.
validator.driver.envwithDRIVER_ROOT=/,NVIDIA_DRIVER_ROOT=/andDISABLE_DEV_CHAR_SYMLINK_CREATION=truemakes it inspect the host root, where the extension put the driver, so it writes its sentinel and the device plugin unblocks. Worth correcting the issue text on one point: the loop onAttempting to validate a driver container installationwas not purely a validator limitation. While the Blackwell schematic was the proprietary one the driver genuinely did not work, and the validator was right to refuse; once the driver worked, the env above was enough. - Gap 4 — no separate spec generation step is needed either.
DEVICE_LIST_STRATEGY=cdi-annotationsplusNVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctklets the device plugin use the host'snvidia-ctkfor the spec, into the/run/cdicontainerd already reads. The pod-recreation caveat from the issue stands and is documented for the website.
One thing the issue does not mention, which fell out of this: the addon default fed
NVreg_NvLinkDisable=1to the operator's driver container through thenvidia-kernel-module-paramsConfigMap, and cozystack passes individual GPUs into worker VMs without the NVSwitches, so without that parameter the driver waits forever for a fabric that cannot come up. With the driver container disabled on Talos the ConfigMap is mounted by nothing and the parameter silently disappears. It now travels throughmachine.kernel.modules— #3571 carries it on the automatic NVIDIA set for that reason.HAMi is not covered. With
addons.hami.enabledthe operator's device plugin stays off and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring on Talos. The production cluster all of this was derived from runs HAMi disabled, so I have no validated configuration for that path and did not guess one. Happy to open a separate issue for it if that is the right home.That accounts for all four gaps in this issue: 1 and the per-group schematic in #3571, 3 and 4 in #3585, and gap 2's remaining half — which schematic to use, and the Blackwell open-modules constraint — as documentation in cozystack/website#643.
- Gap 3 — the validator does have a mode for this after all.
- added a commit that references this issue
on Aug 7, 2026 - added a commit that references this issue
on Sep 21, 2026
Problem / motivation
The 1.6.0 rollover replaced Ubuntu/kubeadm tenant workers with Talos, and the GPU path did not follow: the gpu-operator addon still expects to compile the driver inside the worker VM. Today there is no way to express a working GPU node group through the
KubernetesCR. Found in a production 1.6.0 environment with KubeVirt GPU passthrough, checked against chart source. Four gaps, each hiding the next:machine.kernel.modules. The Talos NVIDIA extensions ship the modules but do not load them, the generated worker machine config has nokernel.modulesblock, and there is no values surface to add one. The only workaround is a hand-writtenTalosConfigTemplate, which survives only because of TalosConfigTemplate is immutable but applied on every reconcile, so machine-config changes never propagate #3515.talos.schematicIDis vanilla, so a GPU node group needs a custom schematic, and nothing documents which one. On Blackwell (GB202,10de:2bb5) onlysiderolabs/nvidia-open-gpu-kernel-modules-productionworks. Withnonfree-kmod-nvidia-productionthe module loads,/dev/nvidia0appears, andnvidia-smi -Lon the host returnsNo devices found, no error anywhere. Neither Cozystack nor Talos docs mention this.driver.enabled: falseandtoolkit.enabled: falsethenvidia-operator-validatorloops on "Attempting to validate a driver container installation" and never writes/run/nvidia/validations/*, so the device plugin stays blocked intoolkit-validation. Past that it fails withfailed to locate libcuda.so:/driver-rootmaps/run/nvidia/driver, which is empty when no driver container ever populated it. Labelling the nodenvidia.com/gpu.deploy.driver=pre-installedor settingNVIDIA_DRIVER_ROOT=/does not help.nvidiacontainerd runtime but does not make it the default, containerd hasenable_cdi = true, and/run/cdiis empty because nothing generates the spec. Tenant pods get the GPU device and no CUDA libraries. CDI injection happens at pod admission, so pods started before the fix must be recreated.Proposed solution
Add a values surface for worker kernel modules. Either declare the NVIDIA set (
nvidia,nvidia_uvm,nvidia_drm,nvidia_modeset) automatically when a node group hasgpus, or take an explicit list. This has to land before or together with #3523: the hash-named template will replace hand-maintained ones and roll workers back to a config without the modules, which silently removes GPUs from running clusters.Document the NVIDIA schematic path (open kernel modules plus container toolkit extensions) with the Blackwell constraint spelled out.
Teach the addon defaults the OS-provided-driver shape: CDI spec generation,
DEVICE_LIST_STRATEGY=cdi-annotations,NVIDIA_DRIVER_ROOT=/, and a replacement for the validator sentinels, so the per-site DaemonSet workaround can be deleted.Docs: debug host first with
chroot /host /usr/local/bin/nvidia-smi -L(Talos has no/bin/shin the host root, invoke binaries directly), and the pod recreation step after fixing a node.Note for implementers: the worker machine config is emitted by both
packages/apps/kubernetesandpackages/apps/kubernetes-nodes, and the render-parity check between them excludes the talos-reconcile Job, so the change must land in both.A working configuration runs in production: open-modules schematic, hand-added
kernel.modules,addons.gpuOperator.valuesOverridewith the CDI settings above, plus a DaemonSet writing the validator sentinels and generating the CDI spec. The reporter offered to implement the kernel modules surface.Related: #3515, #3523.