Repository navigation
Document GPU node groups on Talos workers: kernelModules, per-node-group schematicID, and the Blackwell open-modules requirement #643
Description
Activity
Upstream PR: cozystack/cozystack#3571
The upstream PR grew a second field, and this one needs documenting more than the first, because the failure it fixes is invisible and the diagnosis is counter-intuitive.
schematicIDper node group (nodeGroups.<name>.schematicIDon theKubernetesapp,schematicIDat the root onKubernetes Nodes) overrides the cluster-widetalos.schematicIDfor one node group, applying to both its boot disk image and its Talos installer image.What should be written down, beyond the generated parameter table:
A Talos schematic is a fixed set of system extensions baked into one image, and Talos refuses to finish the boot sequence when an extension service in it cannot start.
ext-nvidia-persistencedandext-nvidia-cdi-genboth require an NVIDIA card. So a node with no GPU that boots an NVIDIA schematic failsstartAllServicesand reboots — roughly every 70 minutes, indefinitely. Sincetalos.schematicIDwas cluster-wide, any cluster mixing GPU and non-GPU node groups had no correct value: the GPU group needs the extensions and every other group is broken by them. The guidance is therefore to set the NVIDIA schematic on the GPU node group only, never cluster-wide, and that sentence is the one worth having on the page.The diagnosis notes matter as much as the guidance, because every obvious signal points the wrong way:
- The node reports
Readyfor the whole cycle. kubelet starts before the failing phase, sokubectl get nodesshows a healthy cluster. - Every pod on the node dies at once (
SandboxChanged,TaintManagerEviction), so all pods on one node carry an identical restart count. That symmetry is the tell. kubectl top nodescan show 96% memory just before the reboot, which reads as an OOM. It is page cache;min_over_time(node_memory_MemAvailable_bytes[30h])never drops near the limit. The discriminant is regularity — several nodes rebooting within seconds of the same uptime is a timer, never memory pressure.- The boot failure is only visible on the serial console (
virtctl console, staying attached across a reboot), and the console is silent between boots.
A useful cross-check to document: compare
dataVolumeTemplates[].spec.source.http.urlon each node group'sKubevirtMachineTemplateagainst theinstall.imageof itsTalosConfigTemplate. If the schematic differs between the two for a group, that group is booting one extension set and would upgrade into another.Related and distinct, worth a cross-reference so readers do not conflate them: a ~70 minute worker reboot cycle can also come from
apidnever obtaining its certificate when port 50001 is missing from the control plane Service. Check that port before concluding it is the schematic.- The node reports
- changed the title
[-]Document GPU node groups on Talos workers: kernelModules field and the Blackwell open-modules schematic requirement[/-][+]Document GPU node groups on Talos workers: kernelModules, per-node-group schematicID, and the Blackwell open-modules requirement[/+]on Aug 6, 2026 Second upstream PR for this: cozystack/cozystack#3585, which changes the GPU Operator addon default for tenant clusters to the OS-provided-driver shape (driver and toolkit disabled, driver validator pointed at the host root, device plugin switched to CDI).
It changes behaviour the docs describe, so it belongs in the same documentation pass as the kernel modules and the schematic:
- Tenant GPU pods get their CUDA libraries through CDI, not through a default
nvidiacontainerd runtime. Worth stating plainly, because it explains the one operational step that surprises people: CDI injection is decided at pod admission, so pods created before a node is working never receive anything and must be recreated. - Operators who worked around the old default by hand, through
addons.gpuOperator.valuesOverride, can drop those overrides once the release carrying #3585 lands. - One override footgun worth a sentence:
mergeOverwritereplaces a list rather than merging it, so an override that adds a single variable todevicePlugin.envsilently drops the CDI defaults and GPUs stop working. The full list has to be restated. - HAMi is explicitly not covered by #3585: with
addons.hami.enabledthe operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring on Talos. If the docs describe HAMi on GPU node groups, that page should not imply the Talos path works yet.
- Tenant GPU pods get their CUDA libraries through CDI, not through a default
Scope update: the upstream PR has been rebased and reduced, and the field paths in this issue narrow with it.
cozystack/cozystack#3571 originally added
kernelModulesandschematicIDto both charts that rendered a worker pool. Upstream has since mergedrefactor(kubernetes): remove worker node pools from the chart, sonodeGroupsno longer exists on theKubernetesapp and the parent-chart half of that PR was dropped.Both fields therefore document as
KubernetesNodesspec fields, at the root, alongsidegpus— not asnodeGroups.<name>.kernelModuleson theKubernetesapp.The substance is unchanged: the three-state behaviour of
kernelModulesstill needs prose because a generated parameter table cannot convey it (the Default column prints[], which is also the explicit opt-out), and the Blackwell/GB202 constraint on which extension the schematic must carry is still documented nowhere.Scope update from cozystack/cozystack#3571: the per-pool
schematicIDfield has been dropped from that PR, because main now scopes the schematic per pool throughosImage.factory.schematicID/osImage.builtin.schematicID(cozystack/cozystack#4171). This issue therefore coverskernelModulesand its three-state behaviour, the Blackwell open-modules requirement, and a GPU pool example that setsosImage.factory.schematicIDto a schematic carrying the NVIDIA extensions, rather than aschematicIDfield.
What is missing
Nothing on the site documents how a GPU node group is meant to work on the Talos workers that Cozystack 1.6.0 introduced for tenant Kubernetes clusters. Two specific gaps, both of which cost real debugging time in a production 1.6.0 environment (upstream cozystack/cozystack#3563):
1. Kernel modules have to be declared. A Talos system extension installs a kernel module but does not load it — that is
machine.kernel.modules' job. Until the upstream PR linked below there was no values surface for it at all; now there is one, and it needs documenting on theKubernetesandKubernetes Nodesreference pages:Kubernetes:nodeGroups.<name>.kernelModulesKubernetes Nodes:kernelModulesat the root, next togpusThe reference tables regenerate from each package's
README.md, so the field descriptions arrive on their own for the apps hardcoded in the websiteMakefile. What does not arrive is the behaviour a prose page has to state: leaving the field unset makes the chart loadnvidia,nvidia_uvm,nvidia_drm,nvidia_modeseton any node group holding annvidia.com/*GPU, a non-empty list replaces that set outright, and an explicit[]opts out entirely. That three-state behaviour is not derivable from a generated parameter table.2. The Blackwell schematic constraint is documented nowhere, and the failure is silent. On Blackwell (GB202, PCI ID
10de:2bb5) the worker schematic must carrysiderolabs/nvidia-open-gpu-kernel-modules-production. With the proprietarysiderolabs/nonfree-kmod-nvidia-productionextension the module loads cleanly,/dev/nvidia0appears, andnvidia-smi -Lon the host then reportsNo devices found— no error in dmesg, in the kubelet log, or anywhere else. Neither the Cozystack docs nor the Talos docs mention this, and there is nothing in the observable state to suggest the extension is the problem. This cost us about an hour; a single sentence on the page would have cost nothing.Related: two debugging notes worth stating wherever the GPU worker path is described, both of which are non-obvious on Talos:
chroot /host /usr/local/bin/nvidia-smi -L. Talos has no/bin/shin the host root, so binaries have to be invoked directly rather than through a shell.Where it belongs
content/en/docs/next/— the reference pages for the two apps, plus whichever GPU/node-group prose page is the right home for the schematic constraint. #561 already asks for GPU passthrough documentation for tenant Kubernetes clusters, but it predates the Talos worker rollover and covers thegpu=onauto-label andNvLinkDisabledefault rather than anything above; the kernel-module surface and the Blackwell extension constraint are new ground. Merge this into #561 if a maintainer prefers to keep the GPU documentation in one place.Worth considering for backport to the released
v1.6documentation directory as well: the Blackwell constraint applies to 1.6.0 as shipped, independently of the upstream PR, since it is a property of the extension rather than of the chart.Upstream
The kernel-modules values surface: see the linked upstream PR. Acknowledged by Aleksei Sviridkin (@lexfrei) as an undocumented behaviour. Filed as the downstream follow-up required by the upstream PR checklist.