feat(gpu-operator): add container variant for preinstalled host driver - #2766

Document the new container variant of cozystack.gpu-operator, paired with
cozystack/cozystack#2766. Covers the apt-installed-driver-and-toolkit
Linux shape that the variant targets: when to pick it over the
passthrough and vGPU variants, prerequisites (host driver + host
nvidia-container-toolkit, validated via nvidia-smi over kubectl debug),
the operator-validator host-driver auto-detect path
(/host/usr/bin/nvidia-smi), Talos caveat with a pointer to the
values-native-talos.yaml reference, install steps, a sample CUDA pod for
verification, the variant comparison matrix, and a cross-reference to
the HAMi sharing guide for tenant Kubernetes clusters.

Lands under operations/ — symmetric with virtualization/gpu.md (VM
passthrough on management cluster) and kubernetes/gpu-sharing.md (HAMi
in tenant Kubernetes addons).

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

Document the new container variant of cozystack.gpu-operator, paired with
cozystack/cozystack#2766. Covers the apt-installed-driver-and-toolkit
Linux shape that the variant targets: when to pick it over the
passthrough and vGPU variants, prerequisites (host driver + host
nvidia-container-toolkit, validated via nvidia-smi over kubectl debug),
the operator-validator host-driver auto-detect path
(/host/usr/bin/nvidia-smi), Talos caveat with a pointer to the
values-native-talos.yaml reference, install steps, a sample CUDA pod for
verification, the variant comparison matrix, and a cross-reference to
the HAMi sharing guide for tenant Kubernetes clusters.

Lands under operations/ — symmetric with virtualization/gpu.md (VM
passthrough on management cluster) and kubernetes/gpu-sharing.md (HAMi
in tenant Kubernetes addons).

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

Document the new container variant of cozystack.gpu-operator, paired with
cozystack/cozystack#2766. Covers the apt-installed-driver-and-toolkit
Linux shape that the variant targets: when to pick it over the
passthrough and vGPU variants, prerequisites (host driver + host
nvidia-container-toolkit registered with containerd via
nvidia-ctk runtime configure, validated with nvidia-smi over
kubectl debug), the host-driver reuse path (driver.enabled=false, so the
operator uses the pre-installed driver at its standard location with no
driverInstallDir override needed on a stock apt install), the Talos
caveat with a pointer to the values-native-talos.yaml reference, install
steps, a sample CUDA pod for verification, the variant comparison
matrix, and a note on why stacking HAMi directly on the container
variant on the management cluster is not a supported combination yet
(both register nvidia.com/gpu).

Lands under operations/ — symmetric with virtualization/gpu.md (VM
passthrough on management cluster) and kubernetes/gpu-sharing.md (HAMi
in tenant Kubernetes addons).

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

Document the new container variant of cozystack.gpu-operator, paired with
cozystack/cozystack#2766. Covers the apt-installed-driver-and-toolkit
Linux shape that the variant targets: when to pick it over the
passthrough and vGPU variants, prerequisites (host driver + host
nvidia-container-toolkit registered with containerd via
nvidia-ctk runtime configure, validated with nvidia-smi over
kubectl debug), the host-driver reuse path (driver.enabled=false, so the
operator uses the pre-installed driver at its standard location with no
driverInstallDir override needed on a stock apt install), the Talos
caveat with a pointer to the values-native-talos.yaml reference, install
steps, a sample CUDA pod for verification, the variant comparison
matrix, and a note on why stacking HAMi directly on the container
variant on the management cluster is not a supported combination yet
(both register nvidia.com/gpu).

Lands under operations/ — symmetric with virtualization/gpu.md (VM
passthrough on management cluster) and kubernetes/gpu-sharing.md (HAMi
in tenant Kubernetes addons).

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

Document the new container variant of cozystack.gpu-operator, paired with
cozystack/cozystack#2766. Covers the apt-installed-driver-and-toolkit
Linux shape that the variant targets: when to pick it over the
passthrough and vGPU variants, prerequisites (host driver + host
nvidia-container-toolkit registered with containerd via
nvidia-ctk runtime configure, validated with nvidia-smi over
kubectl debug), the host-driver reuse path (driver.enabled=false, so the
operator uses the pre-installed driver at its standard location with no
driverInstallDir override needed on a stock apt install), the Talos
caveat with a pointer to the values-native-talos.yaml reference, install
steps, a sample CUDA pod for verification, the variant comparison
matrix, and a note on why stacking HAMi directly on the container
variant on the management cluster is not a supported combination yet
(both register nvidia.com/gpu).

Lands under operations/ — symmetric with virtualization/gpu.md (VM
passthrough on management cluster) and kubernetes/gpu-sharing.md (HAMi
in tenant Kubernetes addons).

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

## What this PR does

Add a new operations guide describing the `container` variant of
`cozystack.gpu-operator` — the architectural mode for containerized GPU
workloads (CUDA pods, ML training, inference) on Linux GPU nodes that
already ship the NVIDIA driver and `nvidia-container-toolkit` via the
distro package manager.

The new page lands at
`content/en/docs/next/operations/gpu-container-workloads.md` and rounds
out the GPU documentation surface:

- [Running VMs with GPU Passthrough](/docs/next/virtualization/gpu/) —
VFIO passthrough of whole GPUs to KubeVirt VMs (`default` variant).
- [GPU Sharing with HAMi](/docs/next/kubernetes/gpu-sharing/) —
fractional GPU sharing in tenant Kubernetes clusters.
- **[Running Containerized GPU
Workloads](/docs/next/operations/gpu-container-workloads/)** — this
page. Containerized GPU workloads on management nodes (`container`
variant).

Content covers when to pick the variant (host driver + host toolkit + a
containerd-registered `nvidia` runtime prerequisite), the host-driver
reuse path (`driver.enabled=false`, so the operator uses the
pre-installed driver at its standard location with no `driverInstallDir`
override on a stock apt install), the Talos caveat with a pointer to the
`examples/values-native-talos.yaml` reference, install steps with
`Package` CR `variant: container`, a sample CUDA pod for verification,
why stacking HAMi directly on this variant is not supported yet, and a
three-row variant comparison matrix.

Companion to cozystack/cozystack#2766, which adds the `container`
variant itself.

### Release note

```release-note
docs(operations): add guide for containerized GPU workloads via the gpu-operator `container` variant.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* New guide for running containerized GPU workloads on cluster nodes:
prerequisites, installation via the Package CR, explicit warning against
using bundles.enabledPackages for this variant, operator health and GPU
allocatable verification, sample CUDA Pod workflow, fractional GPU
sharing via HAMi, and a comparison of container, default (VM
passthrough), and vGPU variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

These four guides were written into cozystack/cozystack under `docs/`
inside the feature PRs that added the code, and never routed anywhere a
user looks. They are operator facing how-tos with no reason to sit in
the source tree, so I moved them here, into both `next/` and `v1.6/`.

### New pages

- `virtualization/vgpu.md` - NVIDIA vGPU on the vgpu variant: two host
driver models, building the vGPU Manager image, SR-IOV profile
assignment, DLS licensing, L40S profile table, OS support.
- `operations/troubleshooting/gpu-operator-host-driver.md` - passthrough
variant refusing to bind vfio-pci when the host already carries an apt
installed driver. This was a 120 line runbook wedged inside the vGPU
guide, it is its own problem so it gets its own page.
- `networking/vm-external-vlan.md` - bridging a VM onto a physically
routed VLAN, and why macvlan does not work with kubevirt bridge binding.
- `networking/hubble.md` - enabling hubble, plus the four dashboards
cozystack ships for it.

Dropped stale content instead of carrying it over: a "Last verified ...
this PR's vgpu variant" stamp, and two claims that the container variant
is unmerged (it landed in cozystack/cozystack#2766 last June, so the
runbook points at it as the no-purge alternative now). Rechecked
NVIDIA/k8s-driver-manager#191, still open.

Two fixes to existing pages while wiring the links.
`virtualization/gpu.md` said vGPU uses mediated devices, that is only
true up to Ampere, Ada Lovelace and newer use SR-IOV virtual functions.
And the hubble dashboards come from `dashboards-infra.list`, which
renders only for tenant-root and cozy-monitoring, so they appear in the
platform grafana and not in a tenant's own.

The CNI plugin staging contract that shared the vm-external-vlan
document stays in cozystack/cozystack as
`packages/system/multus/README.md`, because a bats suite asserts its
prose against the Dockerfile and the manifest. Deletion of the originals
is a separate PR there, i'll link it once it is open.

The container variant landed in #2766 on 2026-06-09. That change
updated the Variants section and the OS support table but left two
passages untouched, both written a fortnight earlier while the variant
was still a pull request. The passthrough host-preparation section
called it "not part of the current package", and the known-limitation
list headed it "not yet merged" under a sentence promising that
neither listed path was available yet. A reader working through the
passthrough purge workaround was told the no-purge alternative did not
exist.

Say instead that the variant ships, and name the part that is easy to
get wrong when reaching for it: bundles.iaas.gpuOperatorVariant
rejects "container" by design, so selecting it goes through
bundles.disabledPackages plus a hand-written Package CR.

The Last verified line at the top of the file carried the same
PR-relative framing for the vgpu variant, which is also a registered
variant now. Name it plainly; on main there is no "this PR" for a
reader to resolve.

The upstream guard the section documents is unchanged. The container
variant sidesteps isHostDriver rather than fixing it, so passthrough
on a host with a pre-installed driver still needs the purge.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The container variant landed in #2766 on 2026-06-09. That change
updated the Variants section and the OS support table but left two
passages untouched, both written a fortnight earlier while the variant
was still a pull request. The passthrough host-preparation section
called it "not part of the current package", and the known-limitation
list headed it "not yet merged" under a sentence promising that
neither listed path was available yet. A reader working through the
passthrough purge workaround was told the no-purge alternative did not
exist.

Say instead that the variant ships, and name the part that is easy to
get wrong when reaching for it: bundles.iaas.gpuOperatorVariant
rejects "container" by design, so selecting it goes through
bundles.disabledPackages plus a hand-written Package CR.

The Last verified line at the top of the file carried the same
PR-relative framing for the vgpu variant, which is also a registered
variant now. Name it plainly; on main there is no "this PR" for a
reader to resolve.

The upstream guard the section documents is unchanged. The container
variant sidesteps isHostDriver rather than fixing it, so passthrough
on a host with a pre-installed driver still needs the purge.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The container variant shipped in #2766 on 2026-06-09. Two passages in
this file still described it as unreleased: the passthrough
host-preparation section called it "not part of the current package",
and the known-limitation list headed it "not yet merged" under a
sentence promising that neither listed path was available yet. A
reader working through the passthrough purge workaround was told the
no-purge alternative did not exist.

The two changes crossed in flight rather than one missing the other.
#2766 updated the document as it then stood. The stale passages were
written on 2026-05-28 while it was still open and landed eleven
minutes behind it, already out of date on arrival.

Say instead that the variant ships, and name the part that is easy to
get wrong when reaching for it: bundles.iaas.gpuOperatorVariant
rejects "container" by design, so selecting it goes through
bundles.disabledPackages plus a Package CR under that same name,
which is how a Package resolves its source.

The Last verified line at the top of the file carried the same
PR-relative framing for the vgpu variant, which is a registered
variant now too. Name it plainly; on main there is no "this PR" for a
reader to resolve.

The upstream guard the section documents is unchanged. The container
variant sidesteps isHostDriver rather than fixing it, so passthrough
on a host with a pre-installed driver still needs the purge.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>