Repository navigation
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant - #2323
Merged
Aleksei Sviridkin (lexfrei) merged 19 commits intoMay 7, 2026
Merged
Conversation
Contributor
📝 WalkthroughWalkthroughAdds vGPU driver variant support to the GPU Operator by introducing a new configuration variant with dedicated Helm values, updating the operator chart from v25.3.0 to v26.3.1, upgrading Node Feature Discovery to v0.18.3, extending CRD schemas for vGPU/CDI capabilities, refactoring templates from gRPC to HTTP probes, and providing end-to-end vGPU deployment documentation and test coverage. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Trivy (0.69.3)Failed to read Trivy output file: ENOENT: no such file or directory, open '/inmem/1272/nsjail-23519067-a499-4b9c-8a3a-cde4ba36e0b7/merged/.trivy-output.json' Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces NVIDIA vGPU support, updates the GPU Operator to version v26.3.0, and upgrades Node Feature Discovery (NFD) to 0.18.3. Key changes include the addition of a vgpu package variant, transition to HTTP-based health probes in NFD, and the adoption of a dedicated utility for CRD management. Feedback suggests updating the documentation to use Secrets for licensing configuration to align with security best practices and removing the deprecated cdi.default field from the ClusterPolicy template.
Comment on lines
+79
to
+104
| 1. Create a ConfigMap with the NLS client configuration in the `cozy-gpu-operator` namespace: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: licensing-config | ||
| namespace: cozy-gpu-operator | ||
| data: | ||
| gridd.conf: | | ||
| ServerAddress=nls.example.com | ||
| ServerPort=443 | ||
| FeatureType=1 | ||
| ``` | ||
|
|
||
| 2. Reference the ConfigMap in the Package values: | ||
|
|
||
| ```yaml | ||
| gpu-operator: | ||
| vgpuManager: | ||
| repository: registry.example.com/nvidia | ||
| version: "550.90.05" | ||
| driver: | ||
| licensingConfig: | ||
| configMapName: licensing-config | ||
| ``` |
Contributor
There was a problem hiding this comment.
The example uses ConfigMap and configMapName for licensing configuration. However, the updated CRDs (nvidia.com_clusterpolicies.yaml and nvidia.com_nvidiadrivers.yaml) mark configMapName as deprecated in favor of secretName. It is recommended to update the documentation to use a Secret for licensing configuration to align with the new best practices for security.
Comment on lines
+152
to
+154
| {{- if .Values.cdi.default }} | ||
| default: {{ .Values.cdi.default }} | ||
| {{- end }} |
Contributor
There was a problem hiding this comment.
The cdi.default field is deprecated in the ClusterPolicy CRD and is no longer used by the GPU Operator (setting cdi.enabled=true now automatically configures it as the default). This block should be removed to align with the updated CRD schema and avoid confusion.
Aleksei Sviridkin (lexfrei)
marked this pull request as ready for review
April 2, 2026 12:51
dosubot
Bot
added
size/XXL
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/gpu-vgpu.md`:
- Around line 81-109: The doc incorrectly instructs creating a ConfigMap and
using driver.licensingConfig.configMapName; update it to create a Secret (kind:
Secret) containing the gridd.conf under stringData and change the package values
example to use driver.licensingConfig.secretName: licensing-config so the
chart/schema picks up the NLS licensing configuration.
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yaml`:
- Around line 97-100: The resources block is being rendered at the Pod spec
level using .Values.master.resources but must be moved into the nfd-master
container spec; update the template so the resources: {{- toYaml
.Values.master.resources }} block is placed inside the container definition for
"nfd-master" (e.g., immediately after that container's fields) rather than after
tolerations, ensuring the resources stanza is nested under the nfd-master
container.
In
`@packages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml`:
- Around line 917-924: The CRD docs for the `operator` field currently list
`Lt`/`Gt` without noting they require the Kubernetes feature gate; update the
`operator` description under the `operator` field (in the NVIDIADriver CRD) to
state that `Lt` and `Gt` require Kubernetes v1.35+ with the
TaintTolerationComparisonOperators feature gate enabled, or alternatively update
the chart's `kubeVersion` constraint to >=1.35.0 so that the chart and the
`NVIDIADriver` spec only allow these operators on supported Kubernetes versions;
pick one approach and make the matching change to either the `operator`
description or the chart `kubeVersion`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 51f9626b-f0b3-498a-82e0-c845bb0b24ad
⛔ Files ignored due to path filters (1)
packages/system/gpu-operator/charts/gpu-operator/Chart.lockis excluded by!**/*.lock
📒 Files selected for processing (32)
docs/gpu-vgpu.mdpackages/core/platform/sources/gpu-operator.yamlpackages/system/gpu-operator/Makefilepackages/system/gpu-operator/charts/gpu-operator/Chart.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/Chart.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/README.mdpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/_helpers.tplpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/master-pdb.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/master.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/nfd-gc-pdb.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/nfd-gc.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/prometheus.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/topologyupdater.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/worker.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/values.yamlpackages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_clusterpolicies.yamlpackages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_nvidiadrivers.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/cleanup_crd.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/clusterpolicy.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/extra-objects.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/nodefeaturerules.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/nvidiadriver.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/operator.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/role.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/rolebinding.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/serviceaccount.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/upgrade_crd.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/validations.yamlpackages/system/gpu-operator/charts/gpu-operator/values.yamlpackages/system/gpu-operator/values-talos-vgpu.yaml
Comment on lines
+97
to
+100
| {{- with .Values.master.resources }} | ||
| resources: | ||
| {{- toYaml . | nindent 8 }} | ||
| {{- end }} |
Contributor
There was a problem hiding this comment.
Resources block is misplaced outside the container spec.
The resources block is rendered at the pod spec level (after tolerations) rather than inside the nfd-master container definition. This would cause a Kubernetes API validation error since resources is not a valid pod spec field.
🐛 Proposed fix to move resources inside container spec
The resources block should be inside the container definition (after line 83, inside the nfd-master container):
args:
- "-prune"
{{- if .Values.master.instance | empty | not }}
- "-instance={{ .Values.master.instance }}"
{{- end }}
+ {{- with .Values.master.resources }}
+ resources:
+ {{- toYaml . | nindent 12 }}
+ {{- end }}
restartPolicy: Never
{{- with .Values.master.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.master.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.master.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
- {{- with .Values.master.resources }}
- resources:
- {{- toYaml . | nindent 8 }}
- {{- end }}
{{- end }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yaml`
around lines 97 - 100, The resources block is being rendered at the Pod spec
level using .Values.master.resources but must be moved into the nfd-master
container spec; update the template so the resources: {{- toYaml
.Values.master.resources }} block is placed inside the container definition for
"nfd-master" (e.g., immediately after that container's fields) rather than after
tolerations, ensuring the resources stanza is nested under the nfd-master
container.
Comment on lines
917
to
924
| operator: | ||
| description: |- | ||
| Operator represents a key's relationship to the value. | ||
| Valid operators are Exists and Equal. Defaults to Equal. | ||
| Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. | ||
| Exists is equivalent to wildcard for value, so that a pod can | ||
| tolerate all taints of a particular category. | ||
| Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). | ||
| type: string |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Do Kubernetes Pod tolerations support LtandGtoperators behind theTaintTolerationComparisonOperators feature gate, and in which Kubernetes versions is that feature available or enabled by default?
💡 Result:
Yes, Kubernetes Pod tolerations support the Lt (Less Than) and Gt (Greater Than) operators behind the TaintTolerationComparisonOperators feature gate. These operators enable numeric comparisons between taint values and toleration values (both must be valid 64-bit positive integers), allowing threshold-based scheduling. The feature is available starting in Kubernetes v1.35 as an alpha feature (disabled by default). It must be explicitly enabled via the --feature-gates=TaintTolerationComparisonOperators=true flag on the API server and kube-scheduler. As of 2026-04-02 (during v1.36 development), it remains alpha and is not enabled by default in any version.
Citations:
- 1: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration
- 2: https://github.com/kubernetes/api/blob/master/core/v1/toleration.go
- 3: https://kubernetes.io/blog/2026/01/05/kubernetes-v1-35-numeric-toleration-operators/
- 4: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
🏁 Script executed:
#!/bin/bash
# Search for Kubernetes version constraints in the repository
fd -t f -e yaml -e txt -e md | xargs grep -l "kubernetes" | head -20
# Also search for version info in common locations
if [ -f "Chart.yaml" ]; then echo "=== Chart.yaml ===" && cat Chart.yaml; fi
if [ -f "requirements.yaml" ]; then echo "=== requirements.yaml ===" && cat requirements.yaml; fi
if [ -f "README.md" ]; then echo "=== README.md (first 50 lines) ===" && head -50 README.md; fi
if [ -f "go.mod" ]; then echo "=== go.mod ===" && cat go.mod; fi
# Search for version constraints in the gpu-operator chart
if [ -d "packages/system/gpu-operator" ]; then
echo "=== Files in gpu-operator ==="
ls -la packages/system/gpu-operator/ | head -20
fiRepository: cozystack/cozystack
Length of output: 10781
🏁 Script executed:
#!/bin/bash
# Search for references to TaintTolerationComparisonOperators or kubernetes version requirements
rg -i "kubernetes|k8s|version|feature.*gate|taint" --type yaml -A 2 -B 2 | head -100Repository: cozystack/cozystack
Length of output: 13930
🏁 Script executed:
#!/bin/bash
# Check if this is a vendored chart by looking for upstream references
if [ -d "packages/system/gpu-operator/charts" ]; then
echo "=== GPU Operator Chart Structure ==="
find packages/system/gpu-operator/charts -type f -name "Chart.yaml" -o -name "values.yaml" | xargs head -20
fiRepository: cozystack/cozystack
Length of output: 2228
Verify Lt/Gt toleration operators are supported in supported Kubernetes versions.
The GPU operator chart supports Kubernetes >= 1.16.0, but the TaintTolerationComparisonOperators feature gate is available only in Kubernetes v1.35+ and is disabled by default. Users can define valid NVIDIADriver specs with Lt/Gt operators that will be rejected by the API server on Kubernetes versions 1.16–1.34 or on v1.35+ without the feature gate explicitly enabled. Either update the CRD documentation to clarify this requirement or update the chart's kubeVersion constraint to reflect the actual compatibility.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@packages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml`
around lines 917 - 924, The CRD docs for the `operator` field currently list
`Lt`/`Gt` without noting they require the Kubernetes feature gate; update the
`operator` description under the `operator` field (in the NVIDIADriver CRD) to
state that `Lt` and `Gt` require Kubernetes v1.35+ with the
TaintTolerationComparisonOperators feature gate enabled, or alternatively update
the chart's `kubeVersion` constraint to >=1.35.0 so that the chart and the
`NVIDIADriver` spec only allow these operators on supported Kubernetes versions;
pick one approach and make the matching change to either the `operator`
description or the chart `kubeVersion`.
Contributor
Testing Report: gpu-operator v26.3.0 on Talos LinuxTested this PR on a Talos v1.12.6 cluster (3× VM nodes with NVIDIA A10 GPUs, Kubernetes v1.34.3). Critical: Talos OS version parsing failureThe ClusterPolicy enters This is a known bug in gpu-operator v26.3.0 (NVIDIA/gpu-operator#2239). The Already fixed upstream in NVIDIA/gpu-operator#2244, released as v26.3.1 on 2026-04-18. The vendored chart should be bumped from v26.3.0 to v26.3.1. This affects all Talos deployments regardless of variant ( Secondary: nvidia-operator-validator permission errorOn nodes where the validator pod was re-rolled, the This happens during SummaryThe primary blocker is the OS version parsing bug — any Talos cluster with GPU nodes will hit it. Bumping the vendored chart to v26.3.1 should resolve this. The validator permission issue needs verification after the bump. |
github-actions
Bot
added
do-not-merge/work-in-progress
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/update-gpu-operator-v26.3.0
branch
from
April 28, 2026 20:05
783b51b to
5a61a33
Compare
github-actions
Bot
added
the
area/virtualization
Aleksei Sviridkin (lexfrei)
changed the title
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
…stack#2323 fixes Mirrors the corrections that landed in cozystack/cozystack#2323 docs/gpu-vgpu.md: * Lead the vgpu install section with an experimental-status alert. The vGPU Manager DaemonSet works, but profile assignment is currently out-of-band (echo <id> > /sys/.../current_vgpu_type per VF) and resets on every reboot — without an automated mechanism the variant silently advertises zero allocatable resources. * Note that the vgpu variant now sets sandboxWorkloads.defaultWorkload to vm-vgpu out of the box (cozystack#2323), so per-node labelling is optional rather than required. * Move imagePullSecrets under vgpuManager: in the example. The chart reads it per-component (vgpuManager.imagePullSecrets, driver.imagePullSecrets, …); placing it at the package root would silently render-no-op and the DaemonSet would ImagePullBackOff without an obvious error. * Split B100 (Blackwell) from L4/L40/L40S (Ada Lovelace) — the driver-model commentary applies to both architectures, but they should be named correctly. * Note that on L40S the SR-IOV VFs share the PF device ID (10de:26b9), so a single pciVendorSelector matches both. Add lspci sanity-check guidance for other generations where PF/VF IDs may differ. * Add a migration note in the Licensing section: chart v25 → v26 deprecated driver.licensingConfig.configMapName in favour of secretName; old key still works but emits a deprecation marker in the CRD. SR-IOV vGPU does not consume the host-side licensing knob at all — relevant only for passthrough operators carrying old configMapName overrides. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
Six fixes from review: * Add a top-of-section warning that the entire vgpu workflow depends on cozystack/cozystack#2323 (still in draft) and KubeVirt v1.9.0 (unreleased) — neither is in any current Cozystack release. Without this banner the page reads like a supported workflow. * Drop the incorrect claim that gpus: expects an mdev resource. The upstream KubeVirt VirtualMachine spec accepts both PCI and mdev resource names under either gpus: or hostDevices:. Restate the hostDevices vs gpus distinction as convention, not a hard constraint, and explain the cozystack/v1alpha1 wrapper trade-off. * Drop the self-contradicting parenthetical that called pciVendorSelector 'the device ID, not the vendor:device pair' — the example right above already uses the vendor:device tuple (10DE:26B9), and the upstream KubeVirt godoc documents the field exactly as that tuple. * Re-quote siderolabs/extensions#461. The issue was closed with stateReason=COMPLETED, not not_planned; the literal phrase 'won't fix' does not appear in the closure. Replace with the substantive reason from rothgar's closing comment. * Fix pre-existing PCI ID error in the passthrough lspci sample output: A10 is 10de:2236 (verified against pci-ids.ucw.cz), the prior text said 10de:26b9 which is L40S. The rest of the passthrough section already uses 2236 correctly, so this was a copy-paste artefact. * Soften the KubeVirt v1.9.0 ETA wording. KubeVirt does not publish hard release dates; replace 'ETA July 2026' with 'targeted for the v1.9.0 release; track the PR for the actual release tag'. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
… to vGPU 20.x Round 4 of review fixes for the vgpu section. Blocking fixes: * Front-matter title was 'Running VMs with GPU Passthrough' but the page now also covers vGPU as a co-equal section. Update title / linkTitle / description / lead paragraph to reflect both flows and add a jump-link to the vGPU section. * Unify ClientConfigToken file mode at 0744 across both sections (step 4 still had 0600, contradicting step 6's 0744 with the NVIDIA-cited justification). 0600 silently breaks license activation when nvidia-gridd runs as a service user. * Harden the profile-loader DaemonSet against copy-paste failures: read-before-write so manual out-of-band changes are visible (and so the kernel does not reject writes while a VM holds the VF and the script logs an error every minute on every busy VF), strip trailing #-comments from ConfigMap lines, log on malformed lines instead of an opaque 'failed to set'. * Add an explicit side-effect callout: while this DaemonSet runs, manual kubectl exec changes to current_vgpu_type are reverted within 60 s. Edit the ConfigMap rather than the sysfs file. * Pin the SR-IOV path to vGPU 20.x explicitly. The previous '17/20' framing was misleading because the SR-IOV / pciHostDevices flow pairs specifically with branch 20 (driver 595.x); the example build is a 595.x .run, and a reader on a 17.x subscription would build the wrong manager image. Worth-fixing: * Reword the cozystack/cozystack#2323 reference time-stable so the text does not need editing once the upstream PR merges. * Tighten the wrapper gpus: claim. The Cozystack VirtualMachine wrapper passes deviceName straight through to KubeVirt; what's missing for SR-IOV vGPU is the hostDevices: surface for headless setups and end-to-end exercise on real hardware. Be precise about which gap is which. * Add (passthrough only) to the licensing migration alert so a vGPU operator does not 'fix' their setup by switching to secretName and then wonder why nothing changed. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Update vendored upstream chart to v26.3.0. Key changes: - CDI enabled by default - Driver version 580.126.20 - NFD subchart updated to 0.18.3 - Component versions bumped (toolkit v1.19.0, device-plugin v0.19.0, dcgm-exporter 4.5.2) - Pin chart version in Makefile for reproducibility Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Add vGPU mode as a new variant alongside the existing passthrough (default) variant. This enables sharing a single physical GPU across multiple VMs using NVIDIA mediated devices. Changes: - Add values-talos-vgpu.yaml with vGPU Manager and vGPU Device Manager enabled - Add 'vgpu' variant to PackageSource - Add documentation for vGPU setup (driver image, NLS licensing, profiles, KubeVirt) Users select the vgpu variant via Package CR and provide their proprietary vGPU Manager image coordinates through component values. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Replace simplified Containerfile with NVIDIA's Makefile-based build system from gitlab.com/nvidia/container-images/driver. The GPU Operator expects pre-compiled kernel modules, not a raw .run file. Add EULA notice about public redistribution of vGPU driver images. Add note about NLS ServerPort being deployment-dependent. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…ording Mirroring the corresponding cleanup in cozystack/website#467 review: * KubeVirt v1.9.0 ETA: drop the 'July 2026' date — KubeVirt does not publish hard release dates and the in-page wording was inconsistent across the file. Replace with 'Targeted at the next minor release (v1.9.0); track the PR for the actual release tag', matching the website doc. * Kernel-headers timing: 'The build downloads kernel headers at container start time' conflated docker build with pod start; the Dockerfile's entrypoint downloads them at runtime. Reword. * Replace guillemets with ASCII quotes for consistency with the rest of the file and the upstream chart docs. * nvidia-smi -q license check: drop the unnecessary -A 1 — License Status value is on the same line as the field name. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
NVIDIA's License System User Guide recommends 744 on the ClientConfigToken file so nvidia-gridd (which does not necessarily run as the file owner) can read it. Previous example used 0600, which can silently break license activation when nvidia-gridd runs as a service user. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Apple Silicon / arm64 build hosts produce arm64 images native; GPU nodes are amd64 and the kubelet pull fails with 'no matching manifest' when the image is built without --platform. Document the flag and add a one-line docker login note for private registries. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…ream NVIDIA's gpu-driver-container repository owns the build path — keeping a parallel set of build instructions in our docs duplicates upstream documentation and goes stale every time the upstream Dockerfile or build args change. Replace the docker build / docker push snippet with a paragraph pointing at the upstream repo and the NVIDIA Licensing Portal as the sources of truth, keep only the items that are Cozystack-specific (private-registry / EULA note). Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Pin a concrete reference set so future readers can tell whether the guide is still accurate against the cluster they're running: KubeVirt main nightly 20260429_74d7c52588, this PR's vgpu variant, NVIDIA vGPU 20.0 host driver 595.58.02, GRID guest driver 595.58.03. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Free-floating 'NVIDIA recommends' claims are flagged as unverifiable in review. Add the doc URL inline so future readers can verify the permission model decision. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/update-gpu-operator-v26.3.0
branch
from
April 30, 2026 11:53
0e2dfad to
b5c1e95
Compare
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
…stack#2323 fixes Mirrors the corrections that landed in cozystack/cozystack#2323 docs/gpu-vgpu.md: * Lead the vgpu install section with an experimental-status alert. The vGPU Manager DaemonSet works, but profile assignment is currently out-of-band (echo <id> > /sys/.../current_vgpu_type per VF) and resets on every reboot — without an automated mechanism the variant silently advertises zero allocatable resources. * Note that the vgpu variant now sets sandboxWorkloads.defaultWorkload to vm-vgpu out of the box (cozystack#2323), so per-node labelling is optional rather than required. * Move imagePullSecrets under vgpuManager: in the example. The chart reads it per-component (vgpuManager.imagePullSecrets, driver.imagePullSecrets, …); placing it at the package root would silently render-no-op and the DaemonSet would ImagePullBackOff without an obvious error. * Split B100 (Blackwell) from L4/L40/L40S (Ada Lovelace) — the driver-model commentary applies to both architectures, but they should be named correctly. * Note that on L40S the SR-IOV VFs share the PF device ID (10de:26b9), so a single pciVendorSelector matches both. Add lspci sanity-check guidance for other generations where PF/VF IDs may differ. * Add a migration note in the Licensing section: chart v25 → v26 deprecated driver.licensingConfig.configMapName in favour of secretName; old key still works but emits a deprecation marker in the CRD. SR-IOV vGPU does not consume the host-side licensing knob at all — relevant only for passthrough operators carrying old configMapName overrides. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
Six fixes from review: * Add a top-of-section warning that the entire vgpu workflow depends on cozystack/cozystack#2323 (still in draft) and KubeVirt v1.9.0 (unreleased) — neither is in any current Cozystack release. Without this banner the page reads like a supported workflow. * Drop the incorrect claim that gpus: expects an mdev resource. The upstream KubeVirt VirtualMachine spec accepts both PCI and mdev resource names under either gpus: or hostDevices:. Restate the hostDevices vs gpus distinction as convention, not a hard constraint, and explain the cozystack/v1alpha1 wrapper trade-off. * Drop the self-contradicting parenthetical that called pciVendorSelector 'the device ID, not the vendor:device pair' — the example right above already uses the vendor:device tuple (10DE:26B9), and the upstream KubeVirt godoc documents the field exactly as that tuple. * Re-quote siderolabs/extensions#461. The issue was closed with stateReason=COMPLETED, not not_planned; the literal phrase 'won't fix' does not appear in the closure. Replace with the substantive reason from rothgar's closing comment. * Fix pre-existing PCI ID error in the passthrough lspci sample output: A10 is 10de:2236 (verified against pci-ids.ucw.cz), the prior text said 10de:26b9 which is L40S. The rest of the passthrough section already uses 2236 correctly, so this was a copy-paste artefact. * Soften the KubeVirt v1.9.0 ETA wording. KubeVirt does not publish hard release dates; replace 'ETA July 2026' with 'targeted for the v1.9.0 release; track the PR for the actual release tag'. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
Apr 30, 2026
… to vGPU 20.x Round 4 of review fixes for the vgpu section. Blocking fixes: * Front-matter title was 'Running VMs with GPU Passthrough' but the page now also covers vGPU as a co-equal section. Update title / linkTitle / description / lead paragraph to reflect both flows and add a jump-link to the vGPU section. * Unify ClientConfigToken file mode at 0744 across both sections (step 4 still had 0600, contradicting step 6's 0744 with the NVIDIA-cited justification). 0600 silently breaks license activation when nvidia-gridd runs as a service user. * Harden the profile-loader DaemonSet against copy-paste failures: read-before-write so manual out-of-band changes are visible (and so the kernel does not reject writes while a VM holds the VF and the script logs an error every minute on every busy VF), strip trailing #-comments from ConfigMap lines, log on malformed lines instead of an opaque 'failed to set'. * Add an explicit side-effect callout: while this DaemonSet runs, manual kubectl exec changes to current_vgpu_type are reverted within 60 s. Edit the ConfigMap rather than the sysfs file. * Pin the SR-IOV path to vGPU 20.x explicitly. The previous '17/20' framing was misleading because the SR-IOV / pciHostDevices flow pairs specifically with branch 20 (driver 595.x); the example build is a 595.x .run, and a reader on a 17.x subscription would build the wrong manager image. Worth-fixing: * Reword the cozystack/cozystack#2323 reference time-stable so the text does not need editing once the upstream PR merges. * Tighten the wrapper gpus: claim. The Cozystack VirtualMachine wrapper passes deviceName straight through to KubeVirt; what's missing for SR-IOV vGPU is the hostDevices: surface for headless setups and end-to-end exercise on real hardware. Be precise about which gap is which. * Add (passthrough only) to the licensing migration alert so a vGPU operator does not 'fix' their setup by switching to secretName and then wonder why nothing changed. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Build job in CI failed because the test relied on bats-specific machinery the in-repo cozytest.sh runner does not provide: * BATS_TEST_DIRNAME is undefined when the file is sourced as a plain shell script (cozytest.sh's mode); switch to BATS_TEST_FILENAME with a $0 fallback, matching hack/check-gpu-recording-rules.bats (REPO_ROOT pattern). * setup()/teardown() are bats lifecycle hooks; cozytest.sh runs each @test in a fresh subshell with set -u and never invokes them. With TMP only assigned in setup(), every test exited immediately on 'TMP: unbound variable'. Inline the mktemp + trap inside each @test so both runners work the same. Verified locally that bash hack/cozytest.sh hack/check-gpu-operator-variants.bats now reports all six tests OK. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
removed
the
do-not-merge/work-in-progress
Aleksei Sviridkin (lexfrei)
marked this pull request as ready for review
May 1, 2026 10:48
Aleksei Sviridkin (lexfrei)
requested a review
from myasnikovdaniil
as a code owner
May 1, 2026 10:48
dosubot
Bot
added
size/L
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request updates the NVIDIA GPU Operator to v26.3.1 and introduces an experimental vGPU variant for SR-IOV support on modern NVIDIA architectures. It refactors existing configuration files to improve clarity and maintainability while ensuring that critical upstream default changes do not negatively impact existing passthrough deployments. The changes include necessary infrastructure for vGPU management, updated DCGM counter definitions, and new validation tests to guard against future configuration regressions. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
dosubot
Bot
added
the
area/platform
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive support for NVIDIA vGPU within the gpu-operator package, adding a new vgpu variant and renaming the existing passthrough configuration. It pins the upstream Helm chart to v26.3.1, updates DCGM metrics, and includes a new BATS test suite to verify variant settings. Feedback was provided to improve the robustness of the test script by checking the Helm template exit code directly to better catch rendering failures.
| --values "$TMP/values.yaml" \ | ||
| --values "$TMP/variant.yaml" \ | ||
| > "$TMP/rendered.yaml" 2>"$TMP/err" | ||
| [ -s "$TMP/rendered.yaml" ] || { cat "$TMP/err" >&2; return 1; } |
Contributor
There was a problem hiding this comment.
While checking for a non-empty file is a good safety measure, it's generally more robust to check the exit code of the helm template command directly to catch rendering errors early.
helm template t "$CHART" \
--values "$TMP/values.yaml" \
--values "$TMP/variant.yaml" \
> "$TMP/rendered.yaml" 2>"$TMP/err" || { cat "$TMP/err" >&2; return 1; }
[ -s "$TMP/rendered.yaml" ]
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
hack/check-gpu-operator-variants.bats (1)
47-56: 💤 Low valueConsider capturing helm exit code for better error diagnostics.
The
render_variantfunction checks for non-empty output, but ifhelm templatefails with a non-zero exit code but produces partial output, the error might be obscured. The stderr redirect helps, but checking$?explicitly would be more robust.Optional: Explicit exit code check
render_variant() { variant="$1" unwrap "$VALUES_DIR/values.yaml" > "$TMP/values.yaml" unwrap "$VALUES_DIR/values-${variant}.yaml" > "$TMP/variant.yaml" - helm template t "$CHART" \ + if ! helm template t "$CHART" \ --values "$TMP/values.yaml" \ --values "$TMP/variant.yaml" \ - > "$TMP/rendered.yaml" 2>"$TMP/err" - [ -s "$TMP/rendered.yaml" ] || { cat "$TMP/err" >&2; return 1; } + > "$TMP/rendered.yaml" 2>"$TMP/err"; then + cat "$TMP/err" >&2 + return 1 + fi + [ -s "$TMP/rendered.yaml" ] || { echo "Empty rendered output" >&2; cat "$TMP/err" >&2; return 1; } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@hack/check-gpu-operator-variants.bats` around lines 47 - 56, The render_variant function currently only verifies that "$TMP/rendered.yaml" is non-empty, which can miss failures when `helm template` exits with a non-zero status but still writes partial output; capture the exit code of the `helm template` invocation (run helm, save its exit code into a variable e.g. rc immediately after the command), and if rc is non-zero, print the saved stderr file ("$TMP/err") and return that non-zero code instead of only checking file size; adjust flow around the `helm template` call inside render_variant so it always checks the command's exit status (and fails fast) while still preserving the stderr output for diagnostics.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yaml`:
- Line 70: The template always emits the imagePullSecrets key even when empty;
update the post-delete-job.yaml around the include
"node-feature-discovery.imagePullSecrets" so the key is rendered only when the
helper returns a non-empty value—e.g., wrap the include call in a conditional
that checks the helper's output (if include
"node-feature-discovery.imagePullSecrets" .) and only then emit
"imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }}";
alternatively, change the helper "node-feature-discovery.imagePullSecrets" to
return the full key:value block only when secrets exist.
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/worker.yaml`:
- Line 39: The template unconditionally emits "imagePullSecrets: {{ include
"node-feature-discovery.imagePullSecrets" . }}" even when the helper returns an
empty string; update the worker.yaml template to first capture the helper output
into a variable (e.g. $ips := include "node-feature-discovery.imagePullSecrets"
.) and wrap the imagePullSecrets block in an if check (if $ips) so the key is
only rendered when non-empty; reference the include
"node-feature-discovery.imagePullSecrets" and the worker.yaml template when
making the change.
---
Nitpick comments:
In `@hack/check-gpu-operator-variants.bats`:
- Around line 47-56: The render_variant function currently only verifies that
"$TMP/rendered.yaml" is non-empty, which can miss failures when `helm template`
exits with a non-zero status but still writes partial output; capture the exit
code of the `helm template` invocation (run helm, save its exit code into a
variable e.g. rc immediately after the command), and if rc is non-zero, print
the saved stderr file ("$TMP/err") and return that non-zero code instead of only
checking file size; adjust flow around the `helm template` call inside
render_variant so it always checks the command's exit status (and fails fast)
while still preserving the stderr output for diagnostics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 40b5be10-213a-49d0-bc93-f89be42e8f64
⛔ Files ignored due to path filters (2)
hack/dcgm-default-counters.csvis excluded by!**/*.csvpackages/system/gpu-operator/charts/gpu-operator/Chart.lockis excluded by!**/*.lock
📒 Files selected for processing (38)
docs/gpu-vgpu.mdhack/check-gpu-operator-variants.batspackages/core/platform/sources/gpu-operator.yamlpackages/system/gpu-operator/Makefilepackages/system/gpu-operator/charts/gpu-operator/Chart.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/Chart.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/README.mdpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/_helpers.tplpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/master-pdb.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/master.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/nfd-gc-pdb.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/nfd-gc.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/prometheus.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/topologyupdater.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/worker.yamlpackages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/values.yamlpackages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_clusterpolicies.yamlpackages/system/gpu-operator/charts/gpu-operator/crds/nvidia.com_nvidiadrivers.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/cleanup_crd.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/clusterpolicy.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/extra-objects.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/nodefeaturerules.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/nvidiadriver.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/operator.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/role.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/rolebinding.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/serviceaccount.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/upgrade_crd.yamlpackages/system/gpu-operator/charts/gpu-operator/templates/validations.yamlpackages/system/gpu-operator/charts/gpu-operator/values.yamlpackages/system/gpu-operator/examples/README.mdpackages/system/gpu-operator/examples/values-native-talos.yamlpackages/system/gpu-operator/values-passthrough.yamlpackages/system/gpu-operator/values-talos.yamlpackages/system/gpu-operator/values-vgpu.yamlpackages/system/gpu-operator/values.yaml
💤 Files with no reviewable changes (1)
- packages/system/gpu-operator/values-talos.yaml
✅ Files skipped from review due to trivial changes (11)
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/README.md
- packages/system/gpu-operator/charts/gpu-operator/templates/rolebinding.yaml
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/prometheus.yaml
- packages/system/gpu-operator/Makefile
- packages/system/gpu-operator/examples/values-native-talos.yaml
- packages/system/gpu-operator/values-passthrough.yaml
- packages/system/gpu-operator/values.yaml
- packages/system/gpu-operator/examples/README.md
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/Chart.yaml
- docs/gpu-vgpu.md
- packages/system/gpu-operator/charts/gpu-operator/templates/serviceaccount.yaml
🚧 Files skipped from review as they are similar to previous changes (8)
- packages/system/gpu-operator/charts/gpu-operator/templates/operator.yaml
- packages/system/gpu-operator/charts/gpu-operator/templates/role.yaml
- packages/system/gpu-operator/charts/gpu-operator/templates/cleanup_crd.yaml
- packages/system/gpu-operator/charts/gpu-operator/Chart.yaml
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/nfd-gc.yaml
- packages/core/platform/sources/gpu-operator.yaml
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/master.yaml
- packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/topologyupdater.yaml
| role: prune | ||
| spec: | ||
| serviceAccountName: {{ include "node-feature-discovery.fullname" . }}-prune | ||
| imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }} |
Contributor
There was a problem hiding this comment.
imagePullSecrets key is always rendered even when empty.
When node-feature-discovery.imagePullSecrets returns an empty string (no secrets configured), this line renders as imagePullSecrets: which is invalid YAML. The key should only be rendered when secrets exist.
🐛 Proposed fix to conditionally render imagePullSecrets
- imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }}
+ {{- with (include "node-feature-discovery.imagePullSecrets" . | fromJson) }}
+ imagePullSecrets:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}Alternatively, modify the helper to include the key in its output.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/post-delete-job.yaml`
at line 70, The template always emits the imagePullSecrets key even when empty;
update the post-delete-job.yaml around the include
"node-feature-discovery.imagePullSecrets" so the key is rendered only when the
helper returns a non-empty value—e.g., wrap the include call in a conditional
that checks the helper's output (if include
"node-feature-discovery.imagePullSecrets" .) and only then emit
"imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }}";
alternatively, change the helper "node-feature-discovery.imagePullSecrets" to
return the full key:value block only when secrets exist.
| imagePullSecrets: | ||
| {{- toYaml . | nindent 8 }} | ||
| {{- end }} | ||
| imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }} |
Contributor
There was a problem hiding this comment.
imagePullSecrets key is always rendered even when empty.
Same issue as in post-delete-job.yaml: when the helper returns an empty string, this renders imagePullSecrets: which is invalid YAML.
🐛 Proposed fix to conditionally render imagePullSecrets
- imagePullSecrets: {{ include "node-feature-discovery.imagePullSecrets" . }}
+ {{- with (include "node-feature-discovery.imagePullSecrets" . | fromJson) }}
+ imagePullSecrets:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@packages/system/gpu-operator/charts/gpu-operator/charts/node-feature-discovery/templates/worker.yaml`
at line 39, The template unconditionally emits "imagePullSecrets: {{ include
"node-feature-discovery.imagePullSecrets" . }}" even when the helper returns an
empty string; update the worker.yaml template to first capture the helper output
into a variable (e.g. $ips := include "node-feature-discovery.imagePullSecrets"
.) and wrap the imagePullSecrets block in an if check (if $ips) so the key is
only rendered when non-empty; reference the include
"node-feature-discovery.imagePullSecrets" and the worker.yaml template when
making the change.
Andrei Kvapil (kvaps)
approved these changes
May 7, 2026
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
Member
There was a problem hiding this comment.
LGTM. Reviewed against 08faa73.
Description is meticulous: every upstream behavior change inherited from the chart v25 → v26 bump (ccManager flip, vgpuDeviceManager flip, licensingConfig.configMapName deprecation, driver.version 570 → 580, vfioManager image switch to k8s-driver-manager, dcgmExporter distroless flavour, NFD subchart moving to oci://registry.k8s.io/nfd/charts) is enumerated in the release-note and reflected in the actual rendered values — verified.
Behavior pinning: values.yaml pins ccManager.enabled and vgpuDeviceManager.enabled back to false with comments explaining why (auto-CC on Hopper would silently change footprint; vgpuDeviceManager would crashloop on Ada/Blackwell because /sys/class/mdev_bus/ does not exist on the vGPU 17/20 driver branch). hack/check-gpu-operator-variants.bats guards both pins plus the per-variant defaultWorkload override, so the next chart bump cannot un-pin them silently. Nice safety net.
values-talos.yaml → values-passthrough.yaml rename is consistent: PackageSource updated, examples/README.md and examples/values-native-talos.yaml comments updated, no dangling references.
docs/gpu-vgpu.md is excellent. Mdev-vs-SR-IOV split is clear, EULA constraints are explicit, profile-assignment caveat ("Do not deploy in production until you have an automated profile-assignment mechanism") is appropriately bold, KubeVirt v1.9.0+ requirement / kubevirt#16890 reference is correct, the L40S PF/VF same-device-ID note has the right "verify on your specific GPU" hedge, and the DLS host-vs-guest licensing distinction is exactly the trap operators usually walk into. "Last verified: 2026-04-29 against KubeVirt main + vGPU 20 host driver 595.58.02 + GRID 595.58.03" makes the doc auditable.
Two non-blocking nits, fine to address in a follow-up:
- values-vgpu.yaml uses repository: "" and version: "" as placeholders. If a user forgets the Package CR override, helm renders an image of ":vgpu-manager:" and the DaemonSet silently doesn't start. Either drop the keys (let upstream defaults surface a clearer error) or wrap with a required helper. Docs already cover this, so it's a small UX cliff at most.
- bats checks rely on grep -A N over rendered YAML — works against current chart structure, would be more durable as yq queries (e.g. yq '.spec.ccManager.enabled' on the ClusterPolicy), but the current form is adequate.
Aleksei Sviridkin (lexfrei)
added a commit
to cozystack/website
that referenced
this pull request
May 7, 2026
## What this PR does Adds a practical guide for running VMs with NVIDIA vGPU on Cozystack to the existing GPU passthrough page. The guide covers the **SR-IOV vGPU path** used by current data-centre GPUs (L4, L40, L40S, B100) on the vGPU 20.x driver branch. The mediated-devices path used by older GPUs (Pascal–Ampere) is explicitly out of scope and the reader is pointed at upstream NVIDIA docs. Steps documented: - Build the proprietary vGPU Manager container image from `github.com/NVIDIA/gpu-driver-container` (the older `gitlab.com/nvidia/container-images/driver` is archived). - Deploy GPU Operator with the `vgpu` variant via Package CR (depends on cozystack/cozystack#2323). - Assign vGPU profiles to SR-IOV VFs (`current_vgpu_type` sysfs), with a periodic profile-loader DaemonSet skeleton and an explicit experimental warning. - Configure DLS licensing via ClientConfigToken (the legacy NLS / `ServerAddress=` / `ServerPort=7070` flow no longer applies). - Patch the KubeVirt CR with `permittedHostDevices.pciHostDevices` (after kubevirt/kubevirt#16890; first stable KubeVirt release with the patch is targeted at v1.9.0). - Sample `VirtualMachine` (raw `kubevirt.io/v1`) using a CDI `DataVolume` so the rootfs has room for in-VM driver install, with a `cloudInitNoCloud` disk that drops the licensing token, `gridd.conf`, an SSH key, and the build dependencies. - vGPU profile reference table for L40S with the Q/A/B suffix taxonomy. - Warning about the 2.4 GiB containerDisk root overflow during in-VM driver install (we observed `SIGBUS` from a write into an mmap of a file the kernel could no longer extend; the new example sidesteps it via `DataVolume`). - Talos is explicitly noted as not recommended for vGPU; passthrough on Talos is unaffected. ### Release note ```release-note NONE ```
1 of 7 tasks
1 of 12 tasks
Aleksei Sviridkin (lexfrei)
added a commit
that referenced
this pull request
Sep 8, 2026
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ## What this PR does Two stale references in `docs/gpu-vgpu.md`. The `Last verified` line says "this PR's `vgpu` variant". The variant ships, so there is no PR left for a reader to resolve that against. The link to `packages/system/gpu-operator/examples/README.md` calls it the flow "that predates the `container` variant". That README now opens with the container variant and sends non-Talos hosts to it. The link is there because it is the Talos path. The rest is gone. #4101 rewrote the same three passages in main, and got right the one thing this PR had wrong: `bundles.iaas.gpuOperatorVariant` takes `container` now, so the `disabledPackages` plus hand-written `Package` route described here is not the way in. The rebase takes main's text for all three and keeps only the two references #4101 did not touch. ### Screenshots Docs only, no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Checked cozystack/website: neither line exists in `content/en/docs/next/virtualization/gpu.md` or `vgpu.md`. The `v1.2` snapshot carries its own PR-relative `Last verified` line, pointing at #2323, but that is a released version's record and out of scope here. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated GPU/vGPU verification details to reference the released vGPU variant. - Documented the available container variant, including deployment guidance and its scope compared with passthrough. - Updated known limitations to reflect that the container variant is shipped and does not configure host-device wiring through KubeVirt. - Clarified when KubeVirt GPU device wiring is applied based on GPU configuration settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an experimental
vgpuvariant for SR-IOV vGPU on Ada Lovelace and newer GPUs.Changes:
values-vgpu.yamlfor the vGPU variant withdefaultWorkload: vm-vgpuso the chart activates the vGPU Manager DaemonSet without per-node labelling.vgpuvariant to PackageSource alongside the existingdefault(passthrough).values-talos.yaml→values-passthrough.yamland pindefaultWorkload: vm-passthroughfor symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically.hack/dcgm-default-counters.csvto match the new dcgm-exporter image tag (4.5.1-4.8.0) — addsDCGM_FI_DEV_FB_RESERVEDand other fields introduced upstream.hack/check-gpu-recording-rules.batspasses against the new fixture.hack/check-gpu-operator-variants.batsthat renders both variants viahelm templateand asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them.The
vgpuvariant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed.vgpuvariant is experimentalNVIDIA's
vgpu-device-managerwalks/sys/class/mdev_bus/, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (echo <id> > /sys/.../current_vgpu_typeper VF) that must be re-applied after every node reboot. Without it,permittedHostDevices.pciHostDevicesreports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR.KubeVirt requirement
KubeVirt advertises VFs via
permittedHostDevices.pciHostDevicesrather thanmediatedDevices. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which ismain-only — released tagsv1.6.x/v1.7.x/v1.8.xdo not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned.Behavior changes inherited from upstream chart v25 → v26
The upstream chart flipped two defaults that are pinned back to
falsehere, sodefault(passthrough) deployments do not silently change footprint on upgrade:ccManager.enabledwas flipped totrueupstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade.vgpuDeviceManager.enabledwas flipped totrueupstream — DaemonSet would render even in passthrough mode.Both stay opt-in via Package CR overrides. The new bats test guards the pin.
Other inherited changes that are user-visible and worth pinning down before upgrade:
driver.licensingConfig.configMapNameis deprecated upstream in favour ofdriver.licensingConfig.secretName. The deprecation marker lives in the CRD'sdescriptionfield (visible viakubectl explain) — there is no helm-render-time warning, the chart template silently preferssecretNameand falls back toconfigMapName. Operators with existing licensing overrides should migrate the key on the next chart upgrade.driver.versionupstream default jumped from570.xto580.x. Invisible on thedefaultvariant (which setsdriver.enabled: false), user-visible for anyone running the upstream Ubuntu driver path.vfioManager.imageswitched fromnvcr.io/nvidia/cuda:12.8.1-base-ubi9tonvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0. Affects everydefault(passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters.dcgmExporter.imageswitched to the-distrolessflavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used tokubectl execinto the exporter for ad-hoc debugging will need to switch tokubectl debugor container ephemeral images.CDI(Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is.node-feature-discoverysubchart source switched from the GitHub Pages Helm repo tooci://registry.k8s.io/nfd/charts. Air-gapped installations need to mirror the new OCI URL.Talos
Talos is not recommended for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (
defaultvariant) on Talos is unaffected.Release note
Summary by CodeRabbit
New Features
Updates