migration 59: one resourceVersion conflict aborts the run, and the Job's three retries are the only fallback #4770

Migration 59 stops at the first VM that changed after it read the VM list, and the hook Job's backoffLimit: 3 is the only retry it has. On a large fleet that may not be enough.

The migration lists all VMs and VMIs once and then patches the VMs that need a copy one by one (packages/core/platform/images/migrations/migrations/59:56-57 and :107-116 at bcda5c8). Each patch starts with a JSON patch test on the metadata.resourceVersion it read in that list (:93). That test keeps the patch from overwriting a newer VM, and I'd keep it. The problem is what happens when it fails. Status writes bump resourceVersion too, and virt-controller writes VM status on its own. Under set -euo pipefail (:39) one failed patch ends the pod.

The Job then starts a new pod (packages/core/platform/templates/migration-hook.yaml:38). That pod reads the version stamp again and lists the VMs again. VMs that were already patched no longer need a copy, so a pod that patches anything before its conflict moves the run forward. But the patches run one after another, so the time between the list and the last patch grows with the number of VMs to patch, and with it the chance of a conflict. Four conflicts in four pods fail the Job, and the platform upgrade stops in its pre-upgrade hook. I don't have numbers on how often this happens. Only VMs whose VMI holds volumes attached by an older kubevirt-csi or by an ephemeral hotplug need a copy at all.

The fix is to handle the conflict per VM: get the VM again right before its patch, compute the patch from that, and on a failed test retry this VM a few times before failing the run.

Raised in the review of #4453: #4453 (review)