Skip to content

VEP 109: Implement vGPU Enabled Live Migration #109

Open
Open
VEP 109: Implement vGPU Enabled Live Migration#109
Assignees
csomani1Aseeef
Labels
kind/enhancementkind/trackersig/compute

Description

@csomani1
csomani1
opened on Oct 27, 2025

Primary contact (assignee):

/assign @csomani1
/assign @Aseeef

Current Feature Stage:

Alpha

Feature Gate:
Depends on: LibvirtHooksServerAndClient
Introduces: VGPULiveMigration

Responsible SIGs:

Primary SIG:
/sig compute

Enhancement link:
#110

Timeline:

Important

Please keep this description up to date.

Activity

  1. csomani1
    added
    kind/enhancement
    kind/tracker
    on Oct 27, 2025
  2. kubevirt-bot
    assigned
    csomani1
    on Oct 27, 2025
  3. kubevirt-bot
    added
    sig/compute
    on Oct 27, 2025
  4. csomani1
    changed the title [-]VEP 108: Implement vGPU Enabled Live Migration[/-] [+]VEP 109: Implement vGPU Enabled Live Migration[/+] on Oct 27, 2025
  5. vladikr
    added this to KubeVirt 1.8 Enhancements Trackingon Oct 29, 2025
  6. vladikr
    moved this to Proposed for consideration in KubeVirt 1.8 Enhancements Trackingon Nov 25, 2025

  7. xpivarc commented on Nov 26, 2025

    @xpivarc
    xpivarc
    on Nov 26, 2025
    Member

    @csomani1 Hi, can you please link a PR of POC if you have it.

  8. csomani1 commented on Nov 26, 2025

    @csomani1
    csomani1
    on Nov 26, 2025
    Author

    csomani1/kubevirt@a7684d2
    This code may be outdated when this PR gets merged, though: kubevirt/kubevirt#16212. I am still reading through the PR

  9. xpivarc
    moved this from Proposed for consideration to Tracked in KubeVirt 1.8 Enhancements Trackingon Dec 19, 2025
  10. iholder101
    moved this from Tracked to Proposed for consideration in KubeVirt 1.8 Enhancements Trackingon Dec 24, 2025
  11. vladikr
    moved this from Proposed for consideration to Tracked in KubeVirt 1.8 Enhancements Trackingon Jan 14, 2026
  12. xpivarc
    added this to KubeVirt 1.9 Enhancements Trackingon Jan 23, 2026
  13. xpivarc
    moved this from Tracked to Exception Required in KubeVirt 1.8 Enhancements Trackingon Feb 25, 2026
  14. vladikr
    assigned
    Aseeef
    on Apr 3, 2026
  15. xpivarc
    moved this to Proposed for consideration in KubeVirt 1.9 Enhancements Trackingon Apr 16, 2026

  16. xpivarc commented on Apr 16, 2026

    @xpivarc
    xpivarc
    on Apr 16, 2026
    Member

    Reviewing this VEP against 1.9 as it missed 1.8:

    Alpha

    • Implement basic functionality and testing.
    • Limitations
      • Users must ensure all worker nodes have identical GPU driver versions since KubeVirt will not take this into account when scheduling the migration
      • KubeVirt is unable to estimate the maximum period for the migration. Use a hard limit that is equal to the existing calculated values (which ignore gpu info)
    • Figure out how to handle any data loss during the migration.

    From these deliverables I am missing the testing part, @Aseeef is this correct? And as well have we progressed with the data loss?

    Further for Beta (1.10?) we will need Docs and update the VEP.

  17. Aseeef commented on Apr 16, 2026

    @Aseeef
    Aseeef
    on Apr 16, 2026
    Member

    I'm not too familiar with what is meant by "Figure out how to handle any data loss during the migration" to be honest. The PR implementation did have basic unit tests. This VEP does need to be updated to include:

    I'll try get to this next week.

  18. xpivarc
    moved this from Proposed for consideration to Tracked in KubeVirt 1.9 Enhancements Trackingon Apr 29, 2026

  19. Barakmor1 commented on Jun 16, 2026

    @Barakmor1
    Barakmor1
    on Jun 16, 2026
    Member

    I'm not too familiar with what is meant by "Figure out how to handle any data loss during the migration" to be honest. The PR implementation did have basic unit tests. This VEP does need to be updated to include:

    I'll try get to this next week.

    Hey @Aseeef any updates regarding this VEP?

  20. Aseeef commented on Jun 16, 2026

    @Aseeef
    Aseeef
    on Jun 16, 2026
    Member

    Hey @Aseeef any updates regarding this VEP?

    The updated graduation requirements merged already: #299

    This VEP AFAIK is all set for v1.9 since its already merged.

  21. xpivarc
    moved this from Tracked to Complete in KubeVirt 1.9 Enhancements Trackingon Jun 26, 2026
  22. xpivarc
    added this to KubeVirt 1.10 Enhancements Trackingon Jul 31, 2026

  23. vladikr commented on Aug 20, 2026

    @vladikr
    vladikr
    on Aug 20, 2026
    Member

    @Aseeef are you planing to work on this in 1.10?

  24. Aseeef commented on Aug 24, 2026

    @Aseeef
    Aseeef
    on Aug 24, 2026
    Member

    @Aseeef are you planing to work on this in 1.10?

    @vladikr subject to hardware availability: maybe. My team has been trying to obtain GPU hardware for months now and struggling. I will post PR updating this VEP to mention the v1.10 target.

  25. xpivarc
    moved this to Proposed for consideration in KubeVirt 1.10 Enhancements Trackingon Sep 2, 2026
  26. vladikr
    moved this from Proposed for consideration to Tracked in KubeVirt 1.10 Enhancements Trackingon Sep 2, 2026

  27. Aseeef commented on Sep 29, 2026

    @Aseeef
    Aseeef
    on Sep 29, 2026
    Member

    One of the requirements for vGPU Live Migration to Beta was: Test migration alongside VEP 248 which added a stall detector. The finding is: it works for most cases except when AllowWorkloadDisruption=False. This case does not work well for QEMU versions below 11.1.

    Here is what goes wrong: since pre-Qemu 11.1 QEMU reports "remaining bytes" to ONLY be host memory remaining. Stall detector uses the remaining bytes to decide when to set to try to "force" a switchover by setting target downtime to max allowed downtime. The stall detector believes that because remaining bytes is very small, setting the switchover should trigger the migration. But in reality, the VFIO device (i.e. the vGPU) still has a lot of data left, so the switchover does NOT trigger. Consequently, stall detector times out the migration.

    In the example I tested, host memory had already long been transferred. But there was still plenty of remaining VFIO memory that hadn't been transferred and isn't reported by QMP (so is invisible to us).

    {"component":"virt-launcher","iteration":4290,"kind":"","level":"info","msg":"Migration info for 464714a4-0f3d-41fb-87a0-da96e1f85f5f: TimeElapsed:296990ms DataProcessed:96075MiB DataRemaining:0MiB DataTotal:32786MiB MemoryProcessed:96073MiB MemoryRemaining:0MiB MemoryTotal:32784MiB MemoryBandwidth:327Mbps DirtyRate:346Mbps Iteration:4290 PostcopyRequests:0 ConstantPages:1608682 NormalPages:11753619 NormalData:45912MiB ExpectedDowntime:0ms DiskMbps:0","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:1000","source":"event","timestamp":"2026-09-29T15:10:55.612945Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4291,"kind":"","level":"info","msg":"Migration info for 464714a4-0f3d-41fb-87a0-da96e1f85f5f: TimeElapsed:296995ms DataProcessed:96075MiB DataRemaining:0MiB DataTotal:32786MiB MemoryProcessed:96073MiB MemoryRemaining:0MiB MemoryTotal:32784MiB MemoryBandwidth:327Mbps DirtyRate:346Mbps Iteration:4291 PostcopyRequests:0 ConstantPages:1608682 NormalPages:11753619 NormalData:45912MiB ExpectedDowntime:0ms DiskMbps:0","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:1000","source":"event","timestamp":"2026-09-29T15:10:55.618125Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4292,"kind":"","level":"info","msg":"stall detected: bestRemainingBytes=0.00Mib outsideWindowMin=0.52Mib candidates=6","name":"vgpu-vm-rhel-2","namespace":"default","pos":"stall-detector.go:251","source":"event","timestamp":"2026-09-29T15:10:55.624871Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4292,"kind":"","level":"info","msg":"convergence action is nothing because: not at a local minima yet","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:642","source":"event","timestamp":"2026-09-29T15:10:55.624891Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4292,"kind":"","level":"info","msg":"Migration info for 464714a4-0f3d-41fb-87a0-da96e1f85f5f: TimeElapsed:297002ms DataProcessed:96075MiB DataRemaining:0MiB DataTotal:32786MiB MemoryProcessed:96073MiB MemoryRemaining:0MiB MemoryTotal:32784MiB MemoryBandwidth:327Mbps DirtyRate:346Mbps Iteration:4292 PostcopyRequests:0 ConstantPages:1608682 NormalPages:11753619 NormalData:45912MiB ExpectedDowntime:0ms DiskMbps:0","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:1000","source":"event","timestamp":"2026-09-29T15:10:55.624903Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4293,"kind":"","level":"info","msg":"max downtime set to 900ms: estimated downtime 0ms within max allowed downtime 900ms","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:665","source":"event","timestamp":"2026-09-29T15:10:55.633961Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}
{"component":"virt-launcher","iteration":4293,"kind":"","level":"info","msg":"Migration info for 464714a4-0f3d-41fb-87a0-da96e1f85f5f: TimeElapsed:297011ms DataProcessed:96076MiB DataRemaining:0MiB DataTotal:32786MiB MemoryProcessed:96074MiB MemoryRemaining:0MiB MemoryTotal:32784MiB MemoryBandwidth:327Mbps DirtyRate:346Mbps Iteration:4293 PostcopyRequests:0 ConstantPages:1608682 NormalPages:11753810 NormalData:45913MiB ExpectedDowntime:0ms DiskMbps:0","name":"vgpu-vm-rhel-2","namespace":"default","pos":"live-migration-source.go:1000","source":"event","timestamp":"2026-09-29T15:10:55.636170Z","uid":"03eeba0a-18e2-4da7-85a6-bfe58c4723c8"}

    Obviously this implies that even though with AllowWorkloadDisruption=True it still "works", we are not making the optimal switchover decisions for that either. In fact, in the case above, we made a bad decision by making the switchover while VFIO memory was still transferring and steadily making progress. And since we have no visibility on how much data is remaining pre-QEMU 11.1, we can't really do anything.

    Conclusion: QEMU 11.1+ is mandatory for having robust vGPU Migrations by leveraging Migration Stall Detection. Otherwise, VGPU Migrations will only reliably work when AllowWorkloadDisruption=True.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/enhancementkind/trackersig/compute

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions