vm-instance: a change to the disk list unplugs hotplug volumes the chart does not render #4769

Since #4453, a vm-instance release that changes its disk list can unplug a hotplug volume from the running VM. This hits any hotpluggable volume that sits in VM.spec but not in .Values.disks: one added with virtctl addvolume --persist, or one that migration 59 copied from the VMI into the VM spec during the upgrade.

The platform now runs KubeVirt with DeclarativeHotplugVolumes. virt-controller rebuilds a running VMI's hotplug volumes from VM.spec and removes every hotpluggable volume the VM spec does not list. Before the gate change it did not reconcile the VMI this way, so an edit to VM.spec.volumes left the VMI alone.

The chart renders spec.template.spec.volumes only from .Values.disks and the cloud-init secret (packages/apps/vm-instance/templates/vm.yaml:139-150 at bcda5c8). Helm patches a VirtualMachine with a JSON merge patch, and a merge patch replaces a list as a whole. While the rendered list is the same in the old and new release, the patch does not touch volumes and the extra entry stays. The first release that changes the list writes the chart's list over the live one. Adding or removing a disk is enough, and so is turning cloud-init or SSH keys on or off. After that, the declarative reconcile detaches the dropped volume from the running VM.

Tenant Kubernetes nodes are not affected. CAPK creates their VirtualMachines from a KubevirtMachineTemplate, and no Helm release renders them.

The upgrade notes in #4453 cover ephemeral hotplug becoming permanent, but not this. The minimum fix is one line in the release notes next to that part. A chart fix is also possible: look up the live VM and render back the hotpluggable volumes that .Values.disks does not list. The cost is that the chart then never removes such a volume. virtctl removevolume still does, since it edits the VM spec.

Raised in the review of #4453: #4453 (review)