Skip to content

Commit 8799a4e

Browse files
lexfreilexfrei
authored
fix(cluster-api): backport live migration before host eviction (#4291)
## What this PR does CAPK v0.1.10 handles host eviction by draining and deleting the guest VMI. It also looks up the physical host name in the guest cluster, and a recreated guest can remain cordoned. Backport CAPK #374, #389 and #392 onto the current provider API and runtime, with separate bounded submission, active migration and fallback windows, and ordinary VMIM objects without a priority, a field the pinned `kubevirt.io/api` v1.5.0 does not carry. The submitted VMIM deadline survives reconciliation; expired, invalid or legacy marker values enter the existing fallback path. Add the manager build to the package pipeline, read its digest pin through the provider imageUrl override, and ship VMIM create permission in the compressed provider manifest. Source tests and four packaging checks pass, including failed-build pin preservation and the rendered ConfigMap payload. The regression tests include unprocessed submissions, legacy and invalid deadline values, repeated reconciles, and a fresh active migration window after submission expiry. The updated manager also cross-compiles for linux/amd64 with Go1.26.4. The previous OCI build predates the submission-deadline correction and does not validate an image containing this change. The updated source still needs the maintainer-owned image build described below. The checked-in pin still identifies the original upstream image, so installing this source chart directly does not enable the backport. The regular main and release builds now include this package, stamp the built image digest, and package that exact artifact; the release workflow also commits the generated pin. Pre-merge E2E validation needs a maintainer-owned build because fork publication only allows image repositories already present in the base tree. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cluster-api): attempt live migration before drain/delete on External eviction, use the guest node name for fallback drain, and uncordon recreated guests. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a Cluster API Provider KubeVirt image pinned to version 0.1.10. - Added migration-first workload eviction with fallback draining and node recovery. - Added permission to create virtual machine migration resources. - Included the provider image in builds and deployment templates. - **Bug Fixes** - Preserved migration deadlines across retries and improved recovery after interrupted or completed migrations. - Added a bounded wait for migrations before falling back to draining. - **Documentation** - Documented the provider image version, patches, build process, and release considerations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 13a2a0f + bc2ba10 commit 8799a4e

11 files changed

Lines changed: 1304 additions & 0 deletions

File tree

‎.gitattributes‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Unified diff context has a marker before Go tabs and blank lines.
2+
packages/system/capi-providers-infraprovider/images/cluster-api-provider-kubevirt/patches/*.diff -whitespace

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ build: build-deps
2929
make -C packages/system/metallb image
3030
make -C packages/system/kamaji image
3131
make -C packages/system/capi-providers-cpprovider image
32+
make -C packages/system/capi-providers-infraprovider image
3233
make -C packages/system/multus image
3334
make -C packages/system/bucket image
3435
make -C packages/system/objectstorage-controller image

‎hack/capk-provider_test.bats‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env bats
2+
# EXIT-TRAP DEBT: 3 -- see hack/bats-no-exit-trap.bats.
3+
# None of the three is debt: each trap sits inside an explicit subshell and
4+
# removes that test's own temp dir, so it does not displace the handler the
5+
# bats binary installs and a failure still prints its `not ok`. The count is
6+
# declared because the ratchet is exact in both directions: it fails if a
7+
# test-level trap is added here, and it fails if one of these is removed.
8+
9+
_capk_fixture() {
10+
mkdir -p "$1/packages/system" "$1/hack" "$1/bin" "$1/oci"
11+
cp -R packages/system/capi-providers-infraprovider "$1/packages/system/"
12+
cp hack/common-envs.mk hack/package.mk "$1/hack/"
13+
cat > "$1/bin/docker" <<'EOF'
14+
#!/bin/sh
15+
set -eu
16+
printf '%s\n' "$@" > "$CAPK_BUILD_ARGS"
17+
[ "${CAPK_BUILD_FAIL:-0}" = 0 ] || exit 77
18+
while [ "$#" -gt 0 ]; do
19+
if [ "$1" = --metadata-file ]; then
20+
shift
21+
printf '{"containerimage.digest":"%s"}\n' "${CAPK_TEST_DIGEST:-sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}" > "$1"
22+
exit 0
23+
fi
24+
shift
25+
done
26+
exit 2
27+
EOF
28+
chmod +x "$1/bin/docker"
29+
}
30+
31+
@test "CAPK compressed provider ships the migration create permission" {
32+
pkg=packages/system/capi-providers-infraprovider
33+
gzip -dc "$pkg/files/components.gz" | cmp - "$pkg/files/infrastructure-components.yaml"
34+
rules=$(gzip -dc "$pkg/files/components.gz" | yq -o=json -I=0 'select(.kind == "ClusterRole" and .metadata.name == "capk-manager-role") | .rules')
35+
printf '%s\n' "$rules" | jq -e '[.[] | select(.apiGroups == ["kubevirt.io"] and .resources == ["virtualmachineinstancemigrations"])] == [{"apiGroups":["kubevirt.io"],"resources":["virtualmachineinstancemigrations"],"verbs":["create"]}]'
36+
}
37+
38+
@test "CAPK rendered provider reads the digest pin outside the gzip" {
39+
(
40+
work=$(mktemp -d)
41+
trap 'rm -rf "$work"' EXIT
42+
pkg=packages/system/capi-providers-infraprovider
43+
helm template capk "$pkg" --namespace cozy-cluster-api > "$work/rendered.yaml"
44+
image=$(yq -r 'select(.kind == "InfrastructureProvider") | .spec.deployment.containers[] | select(.name == "manager") | .imageUrl' "$work/rendered.yaml")
45+
[ "$image" = "$(cat "$pkg/images/cluster-api-provider-kubevirt.tag")" ]
46+
[ "$(yq -r 'select(.kind == "InfrastructureProvider") | .spec.version' "$work/rendered.yaml")" = v0.1.10-infraprovider ]
47+
yq -r 'select(.kind == "ConfigMap") | .binaryData.components' "$work/rendered.yaml" \
48+
| python3 -c 'import base64,gzip,sys; sys.stdout.buffer.write(gzip.decompress(base64.b64decode(sys.stdin.read())))' \
49+
| cmp - "$pkg/files/infrastructure-components.yaml"
50+
)
51+
}
52+
53+
@test "CAPK OCI build stamps a discoverable ref without rewriting components" {
54+
(
55+
work=$(mktemp -d)
56+
trap 'rm -rf "$work"' EXIT
57+
_capk_fixture "$work"
58+
pkg="$work/packages/system/capi-providers-infraprovider"
59+
PATH="$work/bin:$PATH" CAPK_BUILD_ARGS="$work/args" \
60+
make -C "$pkg" image COZYSTACK_VERSION=0 REGISTRY=registry.example.test/cozy \
61+
IMAGE_TAG=pr-test OCI_EXPORT_DIR="$work/oci" PLATFORM=linux/amd64
62+
ref=registry.example.test/cozy/cluster-api-provider-kubevirt:pr-test@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
63+
[ "$(cat "$pkg/images/cluster-api-provider-kubevirt.tag")" = "$ref" ]
64+
grep -Fx -- --push=0 "$work/args"
65+
grep -Fx -- --load=0 "$work/args"
66+
grep -Fx -- "type=oci,dest=$work/oci/cluster-api-provider-kubevirt.oci.tar" "$work/args"
67+
cmp "$pkg/files/components.gz" packages/system/capi-providers-infraprovider/files/components.gz
68+
. hack/lib/image-refs.sh
69+
collect_image_refs "$work/packages" | grep -Fx "$ref"
70+
)
71+
}
72+
73+
@test "CAPK failed builds and missing digests cannot replace the prior pin" {
74+
(
75+
work=$(mktemp -d)
76+
trap 'rm -rf "$work"' EXIT
77+
_capk_fixture "$work"
78+
pkg="$work/packages/system/capi-providers-infraprovider"
79+
pin=$(cat "$pkg/images/cluster-api-provider-kubevirt.tag")
80+
for failure in build digest; do
81+
fail=0
82+
digest=invalid
83+
[ "$failure" != build ] || fail=1
84+
rc=0
85+
PATH="$work/bin:$PATH" CAPK_BUILD_ARGS="$work/args" CAPK_BUILD_FAIL="$fail" CAPK_TEST_DIGEST="$digest" \
86+
make -C "$pkg" image COZYSTACK_VERSION=0 REGISTRY=registry.example.test/cozy \
87+
IMAGE_TAG=pr-test OCI_EXPORT_DIR="$work/oci" > "$work/build.log" 2>&1 || rc=$?
88+
[ "$rc" -ne 0 ]
89+
[ "$(cat "$pkg/images/cluster-api-provider-kubevirt.tag")" = "$pin" ]
90+
done
91+
)
92+
}

‎packages/system/capi-providers-infraprovider/Makefile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,20 @@
11
export NAME=capi-providers-infraprovider
22
export NAMESPACE=cozy-cluster-api
33

4+
include ../../../hack/common-envs.mk
45
include ../../../hack/package.mk
6+
7+
.PHONY: image
8+
9+
UPSTREAM_VERSION = $(shell awk -F= '/^ARG VERSION=/{print $$2}' images/cluster-api-provider-kubevirt/Dockerfile)
10+
COZYSTACK_REV = 0
11+
12+
image:
13+
docker buildx build images/cluster-api-provider-kubevirt \
14+
$(call image-tags,cluster-api-provider-kubevirt,$(UPSTREAM_VERSION)-cozystack.$(COZYSTACK_REV)) \
15+
$(call cache-args,cluster-api-provider-kubevirt) \
16+
--metadata-file images/cluster-api-provider-kubevirt.json \
17+
$(BUILDX_ARGS)
18+
digest=$$(yq -e '."containerimage.digest" | select(test("^sha256:[0-9a-f]{64}$$"))' images/cluster-api-provider-kubevirt.json -o json -r) && \
19+
printf '%s\n' "$(REGISTRY)/cluster-api-provider-kubevirt:$(IMAGE_TAG)@$$digest" > images/cluster-api-provider-kubevirt.tag
20+
rm -f images/cluster-api-provider-kubevirt.json

‎packages/system/capi-providers-infraprovider/files/components.gz‎

0 Bytes
Binary file not shown.

‎packages/system/capi-providers-infraprovider/files/infrastructure-components.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9695,6 +9695,12 @@ rules:
96959695
- get
96969696
- patch
96979697
- update
9698+
- apiGroups:
9699+
- kubevirt.io
9700+
resources:
9701+
- virtualmachineinstancemigrations
9702+
verbs:
9703+
- create
96989704
- apiGroups:
96999705
- kubevirt.io
97009706
resources:

‎packages/system/capi-providers-infraprovider/images/cluster-api-provider-kubevirt.tag‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
quay.io/capk/capk-manager:v0.1.10@sha256:87c2a80413a0f64f65326d379f4905d3f3f2eaf9dab8a6a67a7f7973ca85aef6

‎packages/system/capi-providers-infraprovider/images/cluster-api-provider-kubevirt/Dockerfile‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
FROM --platform=$BUILDPLATFORM golang:1.26.4@sha256:f96cc555eb8db430159a3aa6797cd5bae561945b7b0fe7d0e284c63a3b291609 AS builder
2+
3+
ARG VERSION=v0.1.10
4+
ARG COMMIT=6dbc11c4d0cbfbdcd618566538c10b0d4a07516b
5+
ARG TARGETOS
6+
ARG TARGETARCH
7+
8+
WORKDIR /workspace
9+
10+
RUN curl -fsSL "https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/archive/${COMMIT}.tar.gz" -o /tmp/source.tar.gz \
11+
&& tar -xzf /tmp/source.tar.gz --strip-components=1 \
12+
&& rm /tmp/source.tar.gz
13+
14+
COPY patches /patches
15+
RUN git apply --check /patches/*.diff && git apply /patches/*.diff
16+
17+
RUN --mount=type=cache,target=/go/pkg/mod \
18+
--mount=type=cache,target=/root/.cache/go-build \
19+
go test ./controllers ./pkg/...
20+
21+
RUN --mount=type=cache,target=/go/pkg/mod \
22+
--mount=type=cache,target=/root/.cache/go-build \
23+
CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} \
24+
go build -trimpath -o /manager .
25+
26+
FROM quay.io/capk/capk-manager:v0.1.10@sha256:87c2a80413a0f64f65326d379f4905d3f3f2eaf9dab8a6a67a7f7973ca85aef6
27+
COPY --from=builder --chmod=0755 /manager /manager
28+
USER 65532
29+
ENTRYPOINT ["/manager"]

‎packages/system/capi-providers-infraprovider/images/cluster-api-provider-kubevirt/UPSTREAM.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
The build stays on CAPK v0.1.10, commit `6dbc11c4d0cbfbdcd618566538c10b0d4a07516b`, and preserves its runtime image and API dependencies. The patch backports [guest node lookup #374](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/pull/374), [uncordon after restart #389](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/pull/389), and [migration before drain/delete #392](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/pull/392). It bounds submission waiting to ten minutes, separately from the active migration and drain fallback windows, preserves each deadline across reconciles, and omits migration priority. CAPK v0.1.10 pins `kubevirt.io/api` v1.5.0, whose `VirtualMachineInstanceMigrationSpec` carries only `vmiName`; #392 sets `priority` against v1.9.0, where the field exists. The typed migration from #392 does not compile against the pinned API, which is why the patch builds the migration as an unstructured object and leaves priority to KubeVirt.
2+
3+
`make image` runs the backported regression tests, builds the manager, and writes the resulting tag and digest to `images/cluster-api-provider-kubevirt.tag`. The InfrastructureProvider reads that file through `spec.deployment.containers[].imageUrl`, supported by CAPI operator v0.19.0. The compressed provider manifest contains the additional VMIM create permission. Keeping the image reference out of the gzip lets the existing image collection, promotion, and nightly mirror code see and rewrite it.
4+
5+
The checked-in tag still points at the original upstream image, so applying the chart directly from this source tree does not install the backport. The regular main and release builds call this package's `image` target before packaging, and the release workflow commits the resulting digest pin with its other generated artifacts. A release containing the fix must therefore come from that trusted build path. Fork CI derives its repository allowlist from the base branch, so this new image repository requires a maintainer-owned build for pre-merge E2E validation; changing a fork's allowlist does not authorize publication.

0 commit comments

Comments
 (0)