feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug - #4453

What this PR does

This turns off the deprecated KubeVirt HotplugVolumes gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries !. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. DeclarativeHotplugVolumes is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips handleDeclarativeVolumeHotplug while HotplugVolumes is on, so dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level removevolume on a VM-owned VMI, so the chart now passes --enable-vmi-hotplug-fallback=false. The comments next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a make apply from main before then would crash-loop the CSI controller.

What changes at runtime on an upgrade:

• The VM-level addvolume and removevolume calls now edit VM.spec directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through VM.status.volumeRequests.
• Volumes hot-plugged through the VM subresource are in VM.spec, so they stay attached.
• During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from VM.spec. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted virtualmachineinstances/addvolume), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points cozystack-packages. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with virtctl addvolume without --persist from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of VM.spec and attaches again on a restart, and virtctl removevolume drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level addvolume and removevolume on a VMI that a VM owns (vmiVolumePatch in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from VM.spec, so copying from a stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists:

kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'

Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, migrations.targetVersion, the bats file (its run_migration argument and expected STAMP) and hack/testdata/migration-59-vmi-hotplug.

A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes VM.status.volumeRequests before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it:

kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'

The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated.

A tenant cluster whose kubernetes HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM.

The migration lists the VMs once and patches each one with a resourceVersion test, so a VM that changes in between fails the hook Job. The Job has backoffLimit: 3, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of HotplugVolumes in the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level removevolume (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster.

hack/migration-59-vmi-hotplug-volumes.bats runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept HotplugVolumes in disabledFeatureGates and refuse DeclarativeHotplugVolumes there. The CSI test pins the flag at false, and the make update idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test.

Screenshots

No UI change.

Downstream repositories

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.

Summary by CodeRabbit

• New Features
  • Declarative hotplug volumes are enabled by default for KubeVirt.
  • Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated.
• Bug Fixes
  • The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail.
  • The Kubernetes CSI driver no longer uses the VMI hotplug fallback.