Enabling VM import doesn't work today because the forklift operator image cannot be pulled. packages/system/forklift-operator/templates/forklift-operator.yaml pins 16 quay.io/kubev2v images by digest. For 14 of them, the operator image included, quay returns MANIFEST_UNKNOWN. The package is opt-in (cozystack.forklift-operator in bundles.enabledPackages). Nothing fails until someone turns it on.
I checked every pinned ref in the file with crane manifest <ref> on 2026-09-26:
- Missing, all pinned as
release-2.11@sha256:...: forklift-operator, forklift-controller, forklift-api, forklift-validation, forklift-virt-v2v, forklift-virt-v2v-xfs, forklift-cli-download, populator-controller, ovirt-populator, openstack-populator, vsphere-xcopy-volume-populator, forklift-ova-provider-server, forklift-hyperv-provider-server, forklift-ova-proxy.
- Still resolving:
forklift-must-gather:v2.7.1 and forklift-console-plugin:release-v2.6.7.
The problem is the tag. release-2.11 is a branch tag that upstream moves. Today it points at sha256:29d843a9… for the operator, and the old digest we pinned is gone from quay. The 2.11 line has no immutable tag: crane ls quay.io/kubev2v/forklift-operator shows only release-2.10 and release-2.11 for it. The pins came in with 54621bc on 2026-09-07 and lasted less than three weeks. Repinning to the current release-2.11 digests works until the tag moves again, and then it breaks the same way.
This is the same failure as #3305, which covers Dockerfile base images. These refs live in a vendored template, though, and a check over FROM lines would not see them.
I see two fixes that last. One is to copy the images to ghcr.io/cozystack and pin the copies. The other is to move to an upstream line that publishes immutable version tags. In both cases, a check that resolves every pinned digest in deployed manifests would have caught this the day it broke.
Enabling VM import doesn't work today because the forklift operator image cannot be pulled.
packages/system/forklift-operator/templates/forklift-operator.yamlpins 16quay.io/kubev2vimages by digest. For 14 of them, the operator image included, quay returnsMANIFEST_UNKNOWN. The package is opt-in (cozystack.forklift-operatorinbundles.enabledPackages). Nothing fails until someone turns it on.I checked every pinned ref in the file with
crane manifest <ref>on 2026-09-26:release-2.11@sha256:...: forklift-operator, forklift-controller, forklift-api, forklift-validation, forklift-virt-v2v, forklift-virt-v2v-xfs, forklift-cli-download, populator-controller, ovirt-populator, openstack-populator, vsphere-xcopy-volume-populator, forklift-ova-provider-server, forklift-hyperv-provider-server, forklift-ova-proxy.forklift-must-gather:v2.7.1andforklift-console-plugin:release-v2.6.7.The problem is the tag.
release-2.11is a branch tag that upstream moves. Today it points atsha256:29d843a9…for the operator, and the old digest we pinned is gone from quay. The 2.11 line has no immutable tag:crane ls quay.io/kubev2v/forklift-operatorshows onlyrelease-2.10andrelease-2.11for it. The pins came in with 54621bc on 2026-09-07 and lasted less than three weeks. Repinning to the currentrelease-2.11digests works until the tag moves again, and then it breaks the same way.This is the same failure as #3305, which covers Dockerfile base images. These refs live in a vendored template, though, and a check over
FROMlines would not see them.I see two fixes that last. One is to copy the images to
ghcr.io/cozystackand pin the copies. The other is to move to an upstream line that publishes immutable version tags. In both cases, a check that resolves every pinned digest in deployed manifests would have caught this the day it broke.