Skip to content

vm-instance does not return to Ready across a platform upgrade (passes on fresh install) #3984

Closed
Closed
vm-instance does not return to Ready across a platform upgrade (passes on fresh install)#3984
Labels
area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

Description

@myasnikovdaniil
myasnikovdaniil
opened on Aug 28, 2026

What happens

A vm-instance VirtualMachine that exists before a platform upgrade does not come back within its chart's timeout after one. The HelmRelease reports:

Released=False reason=UpgradeFailed: Helm upgrade failed for release tenant-test/vm-instance-test
with chart [email protected]+16d84947b222:
timeout waiting for: [VirtualMachine/tenant-test/vm-instance-test status: 'InProgress']

The VM sits at InProgress for the full 10 minutes the chart waits, and the release is still not Ready 20 minutes later when the platform readiness gate gives up.

Why this is upgrade-specific

Both halves were observed in the same CI run, on the same commit and the same runner class, which is what makes this worth filing rather than treating as a flake:

  • fresh install, in-tree suite: --- PASS: chainsaw/vminstance (164.20s) — the identical workload comes up in under three minutes
  • across an upgrade, v1.6.2 → main: the release above times out

So whatever this is, it is reached by upgrading a cluster that already runs the VM, not by creating one.

Evidence from the window, and what it does not prove

Two KubeVirt-side restarts fall inside the upgrade window, both worth checking before anything else:

cozy-kubevirt-cdi/cdi-uploadproxy  (restarts=1)
  F uploadproxy.go:104] Unable to create certwatcher: tls: failed to find any PEM data in certificate input
cozy-kubevirt/virt-api             (restarts=1)

I am deliberately not claiming these are the cause. cdi-uploadproxy is not on the path that starts an existing VM, and a virt-api restart during a component upgrade is expected; the reason they are here is that a VM stuck at InProgress while its control plane is rolling is the shape those two would produce, and nothing else in the captured state pointed anywhere. The run's crust-gather snapshot for the failing suite is attached to the job's upgrade-cozyreport artifact and holds the VirtualMachine, VMI and virt-launcher state at failure time, which is where a root cause should come from.

Where it was found

The release upgrade E2E lane (#3276), on the v1.6.2 → main path: job log. That lane is advisory and gates nothing. While this is open it is recorded as the one expected not-Ready release in the lane's ledger, so a NEW upgrade failure is still distinguishable from this one; the ledger entry fails the lane if this release ever comes back Ready, so it cannot outlive the fix.

Activity

  1. myasnikovdaniil
    added
    kind/bugCategorizes issue or PR as related to a bug
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)
    on Aug 28, 2026
  2. github-actions
    added
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Aug 28, 2026

  3. dosubot commented on Aug 28, 2026

    @dosubot
    dosubotbot
    on Aug 28, 2026 – with DosuBot

    myasnikovdaniil This looks like the same failure mode already tracked in #3734 ("vm-instance releases never converge: kstatus observedGeneration pre-check gates healthy KubeVirt VMs"), with a mitigation open in PR #3735.

    The root cause identified there: since helm-controller v1.5.0, the vm-instance HelmRelease is health-assessed by the kstatus poller, whose CEL status evaluator has a built-in pre-check — if status.observedGeneration trails metadata.generation, it returns InProgress before any other logic runs, and this pre-check cannot be overridden by healthCheckExprs [1]. KubeVirt legitimately keeps a VirtualMachine's observedGeneration behind generation whenever a spec.template change is pending a restart, a VM is stopped, or via the open upstream bug kubevirt/kubevirt#18700 [1]. So a healthy, running VM reads as permanently InProgress, and the release times out with exactly the timeout waiting for: [VirtualMachine/... status: 'InProgress'] message you quoted.

    This explains the upgrade-vs-fresh-install split: on a fresh install the VM's generation and observedGeneration start aligned, but a VM that already existed before the upgrade is likely to carry a stale observedGeneration. Specifically, observedGeneration is sourced from the VMI annotation kubevirt.io/vm-generation, which conditionallyBumpGenerationAnnotationOnVmi only bumps when the running VMI's spec exactly matches the ControllerRevision it started from — any unactioned template change leaves it stale indefinitely [2].

    The mitigation in #3735 sets release.cozystack.io/helm-install-disable-wait: "true" on the vm-instance ApplicationDefinition so the HelmRelease no longer waits on VirtualMachine readiness at all, restoring pre-1.5.0 behavior. An independent v1.6.1 lab confirmed this fix, with disk hotplug operations dropping from ~7 min to ~40 s and no further failed revisions [2]. That same report noted that existing HelmReleases only pick up the new spec on their next write through the apps API, and that cozystack-api may need a manual restart since the annotation isn't hashed into config-hash [2].

    Given this closely matches your report (same error string, same upgrade-vs-fresh-install split, same InProgress stall), the virt-api/cdi-uploadproxy restarts you observed are likely a red herring — the kstatus/observedGeneration gap is the more probable driver. You could confirm by checking whether your CI run's VM had observedGeneration < generation before the upgrade kicked in.

    To reply, just mention @dosu.

    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about cozystack Share Dosu with your team

  4. myasnikovdaniil
    added a commit that references this issue on Aug 28, 2026

    test(e2e): record the upgrade lane's known failure instead of staying…

    c7dca5f

  5. crossRT commented on Aug 31, 2026

    @crossRT
    Ray Low (crossRT)
    on Aug 31, 2026
    Contributor

    i encounter this exactly same issue when upgrading cozystack from v1.6.1 to v1.6.2.

    i saw this message saying the VirtualMachine status stuck on InProgress.
    but i checked it's actually Ready, and failed to figure out what causing this.

    Helm upgrade failed for release tenant-xxx/vm-instance-forgejo-runner with chart [email protected]+47520a2b8bd9: timeout waiting for: [VirtualMachine/tenant-xxx/vm-instance-forgejo-runner status: 'InProgress']

    here's the helm logs.

    Last Helm logs:

2026-08-31T02:14:18.717928824Z: using server-side apply for resource update: {"dryRun":false,"fieldValidationDirective":"Strict","forceConflicts":true,"upgradeClientSideFieldManager":false}
2026-08-31T02:14:18.717932912Z: checking resources for changes: {"resources":7}
2026-08-31T02:14:18.740310642Z: Patched resource: {"gvk":"/v1, Kind=Secret","name":"vm-instance-forgejo-runner-ssh-keys","namespace":"tenant-xxx"}
2026-08-31T02:14:18.802358213Z: Patched resource: {"gvk":"/v1, Kind=Secret","name":"vm-instance-forgejo-runner-cloud-init","namespace":"tenant-xxx"}
2026-08-31T02:14:18.838433297Z: Patched resource: {"gvk":"rbac.authorization.k8s.io/v1, Kind=Role","name":"vm-instance-forgejo-runner-dashboard-resources","namespace":"tenant-xxx"}
2026-08-31T02:14:18.856545124Z: Patched resource: {"gvk":"rbac.authorization.k8s.io/v1, Kind=RoleBinding","name":"vm-instance-forgejo-runner-dashboard-resources","namespace":"tenant-xxx"}
2026-08-31T02:14:18.904365803Z: Patched resource: {"gvk":"/v1, Kind=Service","name":"vm-instance-forgejo-runner","namespace":"tenant-xxx"}
2026-08-31T02:14:19.061429242Z: Patched resource: {"gvk":"kubevirt.io/v1, Kind=VirtualMachine","name":"vm-instance-forgejo-runner","namespace":"tenant-xxx"}
2026-08-31T02:14:19.117309846Z: Patched resource: {"gvk":"cozystack.io/v1alpha1, Kind=WorkloadMonitor","name":"vm-instance-forgejo-runner","namespace":"tenant-xxx"}
2026-08-31T02:15:07.773891414Z: warning: upgrade failed: {"error":{},"name":"vm-instance-forgejo-runner"}
2026-08-31T02:15:07.810310987Z: warning: upgrade failed: {"error":{},"name":"vm-instance-forgejo-runner"}
  Warning  UpgradeFailed  2m50s  helm-controller  Helm upgrade failed for release tenant-xxx/vm-instance-forgejo-runner with chart [email protected]+47520a2b8bd9: timeout waiting for: [VirtualMachine/tenant-xxx/vm-instance-forgejo-runner status: 'InProgress']

  6. crossRT commented on Aug 31, 2026

    @crossRT
    Ray Low (crossRT)
    on Aug 31, 2026
    Contributor

    updates: i resolved this issue by shutting down the VM, wait the HR reconcile again.
    and the HR status will back to normal.

  7. lexfrei commented on Sep 2, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Sep 2, 2026
    Contributor

    Same mechanism as #3734: the vm-instance ApplicationDefinition has no release.cozystack.io/helm-install-disable-wait, so the kstatus observedGeneration pre-check gates the release. Stopping the VM realigns generation and observedGeneration, which is why the workaround in the last comment works. Closing as a duplicate; the fix is PR #3735.

  8. Aleksei Sviridkin (lexfrei)
    closed this as completedon Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions