vm-instance releases never converge: kstatus observedGeneration pre-check gates healthy KubeVirt VMs #3734

Activity

1. github-actions

   added

   triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

   on Aug 12, 2026
2. lfinmauritius commented on Aug 28, 2026

   

   lf (lfinmauritius)

   on Aug 28, 2026

   Contributor

   More actions

   Independent confirmation from a Cozystack v1.6.1 lab (Talos v1.13.6, KubeVirt v1.8.4, 3 combined control-plane/worker nodes, 3 VMInstances), 2026-08-28.

   Symptom, measured. Every VMInstance HelmRelease was in the loop upgrade → kstatus wait → 10 min timeout → failed → retry (75 / 100 / 137 helm revisions on our three machines). Because helm-controller serialises upgrades per release, any change made while an upgrade is waiting (e.g. a disk attach through the apps API) only gets applied once the running one times out: 3–15 min per gesture, while the declarative volume hotplug itself completes in ~30 s once the VirtualMachine is patched.

   Root cause, confirmed on the code. status.observedGeneration comes from the VMI annotation kubevirt.io/vm-generation, which conditionallyBumpGenerationAnnotationOnVmi (pkg/virt-controller/watch/vm/vm.go, v1.8.4) only bumps when the whole current spec.template equals the ControllerRevision the VMI was started from. Any template change not followed by a restart therefore leaves observedGeneration < generation, and the kstatus pre-check in fluxcd/pkg/runtime/cel returns InProgress before any healthCheckExprs is evaluated. We also verified that template labels (e.g. helm.sh/chart: <digest>) are not the trigger: KubeVirt neither sets RestartRequired for them (syncRestartRequired compares Template.Spec only) nor syncs them to the VMI.

   Mitigation from #3735, applied and measured. With release.cozystack.io/helm-install-disable-wait: "true" on the vm-instance ApplicationDefinition (v1.6.1 cozystack-api already honours it, same mechanism as kubernetes-rd), after regenerating the HelmReleases: attach of a ready disk to a running VM — HelmRelease deployed 10 s after the API call, VMI volume Ready at 41 s (was 7 min 20 s); detach — volume removed at 49 s (was 5 min); no failed revisions since; RestartRequired never set; VM health read from the VirtualMachine as expected.

   Two operational notes for the PR description:

   1. The annotation is read by cozystack-api at start-up, and ApplicationDefinitionReconciler computes cozystack.io/config-hash over Name + Spec only — an annotation-only change to the ApplicationDefinition does not roll cozystack-api. On an existing cluster the annotation had no effect until a manual kubectl -n cozy-system rollout restart deployment/cozystack-api (we waited 6 min to be sure). Either hashing metadata.annotations too, or documenting the restart, would avoid a silent no-op.
   2. As stated in the PR, existing HelmReleases pick the new spec up only on the next write of their Application. In our case a stop/start through the apps API was enough (the stop regenerated the HelmRelease with DisableWait; the queued upgrade still had to wait for the in-flight one to time out once, then everything converged in seconds).

   Happy to share the exact timelines if useful.
3. ghostrider0470 commented on Sep 18, 2026

   

   Abdagić Hamza (ghostrider0470)

   on Sep 18, 2026

   More actions

   What changes the template on an upgrade that changes nothing: the chart pins
   the kube-ovn address into it, from a lookup that has nothing to find at
   install time.

   packages/apps/vm-instance/templates/vm.yaml looks up the VM's kube-ovn IP
   object and, when it exists, writes the address and MAC into
   spec.template.metadata.annotations:

   {{- $ovnIP := lookup "kubeovn.io/v1" "IP" "" $ovnIPName }}
   {{- if $ovnIP }}
   ovn.kubernetes.io/mac_address: {{ $ovnIP.spec.macAddress | quote }}
   ovn.kubernetes.io/ip_address: {{ $ovnIP.spec.ipAddress | quote }}
   {{- end }}

   The IP object is created when the VM's pod starts, so the install renders
   without these annotations and the first upgrade after boot adds them. That
   makes the first upgrade of every VM a spec.template change, whatever the
   upgrade was for, which is the case the comment above describes. On v1.6.2 the
   template diff of such an upgrade was only these two lines:

              "kubevirt.io/pci-topology-version": "v3",
   +          "ovn.kubernetes.io/ip_address": "<the VM's address>",
   +          "ovn.kubernetes.io/mac_address": "<the VM's MAC>"

   and the VirtualMachine then sat at generation 2, observedGeneration 1, desiredGeneration 2 for the whole 10-minute timeout while Running and
   Ready=True.

   So #3735 fixes the wait, and this is what triggers it. Rendering the same
   template on install and upgrade would stop a routine upgrade (a platform chart
   bump, a resize, a metadata-only write through the apps API) from being a
   template change at all. That could mean leaving the pin out of the template, or
   setting it on the VM at a point where install and upgrade agree. Which is right
   depends on why the address is pinned: presumably so it survives the pod being
   recreated, which is worth keeping.

   One thing that makes it fire more often than it would: cozystack-api applies
   spec defaults on read but not on create, so the first write of any kind to a
   VMInstance, even an annotation, upgrades its release. Reported as #4342.
4. x-itm commented on Sep 19, 2026

   

   X-ITM (x-itm)

   on Sep 19, 2026

   More actions

   The lookup makes the rendered VM spec depend on live cluster state, so it creates a second problem beyond the kstatus gate: an otherwise no-op upgrade can become a spec.template mutation after the IP object appears. Disabling the Helm readiness wait addresses the immediate release deadlock, but it would also be worth removing or stabilising this lookup-driven template change. In particular, avoid rendering empty IP/MAC annotations and compare helm get manifest before/after the first boot to identify every generated template diff. CEL health expressions cannot solve this while the observedGeneration pre-check runs first.
5. Aleksei Sviridkin (lexfrei)

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)Issues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)

   triage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on

   priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next release

   and removed

   triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

   on Sep 23, 2026
6. Aleksei Sviridkin (lexfrei)

   closed this as completedin #3735on Sep 24, 2026
7. pull

   added a commit that references this issue on Sep 24, 2026

   fix(vm-instance): disable helm readiness wait on the VMInstance relea…

   ...

   15035a9