Skip to content

[Backport release-1.6] fix(kubeovn): retry VM migration setup after scheduling - #4328

Merged
myasnikovdaniil merged 4 commits into
release-1.6from
backport-4253-to-release-1.6
Sep 25, 2026
Merged

myasnikovdaniil merged 4 commits into
release-1.6from
backport-4253-to-release-1.6

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Description

Backport of #4253 to release-1.6, together with #3460 which it depends on. On 1.6 kube-ovn chart and image still come from cozystack/kubeovn-chart, so draft Dockerfile had nothing to build it and kubeovn build failed (no image target). #3460 moves build in-tree on same kube-ovn 1.15.10 this branch runs, and brings nightly-mirror fix for kube-ovn split registry host that becomes needed once kube-ovn is built by CI. Bot draft also dropped second commit of #4253 (fix-vmim-target-pod-retry.diff), it is back now.

I built image locally, all three patches apply to kube-ovn v1.15.10 and TestVMIMigrationSchedulingRetry passes.

@github-actions

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-4253-to-release-1.6
git worktree add --checkout .worktree/backport-4253-to-release-1.6 backport-4253-to-release-1.6
cd .worktree/backport-4253-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x 55a2918fcbe894982634280fdf543da4a4feb7a6 6e904470910e4ec627cfb4098c8d8b7a08011897
git push --force-with-lease

@coderabbitai

coderabbitai Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 8dfac6f3-0b7c-4162-ae95-a075ddb4988d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@yankawai

Copy link
Copy Markdown
Contributor

This one is not a conflict to resolve — release-1.6 has nowhere for the change to land.

The red check is the first symptom: make -C packages/system/kubeovn image exits 2 with "No rule to make target 'image'" (run 35350652218, job 105618002231). On this line the package is Chart.yaml, Makefile, values.yaml and one test; its update target pulls chart, values and Chart.yaml from cozystack/kubeovn-chart, and the whole images/kubeovn layout — Dockerfile, patches, image target — is main-only. So the branch adds a Dockerfile and a patch beside a Makefile that cannot build them, and the root Makefile line it adds calls a target that does not exist.

Two other things happened on the way: the branch carries one of the two patches from #4253 — fix-vmim-target-pod-retry.diff is missing — and a 92-line test file against the 100 on main.

Green CI would not be enough either. 1.6 runs the image its chart repo pins, ghcr.io/cozystack/cozystack/kubeovn:v1.15.10@sha256:741299cb…, and cozystack/kubeovn-chart is archived, last pushed 2026-07-27. Nothing in this tree rebuilds that image for the 1.6 line, so a patch file added here would sit unused.

The bug itself is present on 1.6: both lines run kube-ovn 1.15.10.

Three routes, and the choice is yours:

  1. leave the fix to the next minor and close this PR;
  2. unarchive kubeovn-chart, add both patches to its docker/ and patches/ trees, rebuild and re-pin the digest in the 1.6 values;
  3. port the images/kubeovn layout from main onto release-1.6 — Dockerfile, both patches, the image target and the root Makefile line — which is the larger change but leaves 1.6 where main already is.

I can prepare 3 as a branch if that is the direction. 2 needs write access to an archived repository, which I do not have.

@yankawai

Copy link
Copy Markdown
Contributor

One correction to route 3, after #4329 was filed yesterday: a fork PR that adds a new first-party image cannot get a green E2E Tests today. The push allowlist is derived from the base tree, so an image the PR itself adds is missing by construction and the job refuses it. #4291 hit exactly that.

So porting the images/kubeovn layout onto release-1.6 from my fork would build and then fail the same push step. Until #4329 is resolved, route 3 is open to a branch inside this repository or it waits for the CI fix. Routes 1 and 2 are unaffected.

@myasnikovdaniil
myasnikovdaniil force-pushed the backport-4253-to-release-1.6 branch from 97fd1a1 to 92ba38c Compare September 23, 2026 11:52
@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Sep 23, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review September 23, 2026 12:03
myasnikovdaniil added a commit that referenced this pull request Sep 24, 2026
…pping the pod range (#4432)

Backport of #3750 to `release-1.6`.

Sandbox Talos config and platform install on this branch still both use
`10.244.0.0/16`, so kube-ovn and cilium allocate from one range and
cilium refuses pod sandboxes with `IP ... is already in use`. It hit
`mariadb-2-backup-roundtrip` on the E2E run of #4328 here.

Cherry-picked clean with `-x`.

### Testing

- `make unit-tests` green, POSIX sh sweep clean.
- Real check is E2E on this PR.

```release-note
NONE
```
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-4253-to-release-1.6 branch from 92ba38c to 6e1ecbc Compare September 24, 2026 14:49
myasnikovdaniil added a commit that referenced this pull request Sep 25, 2026
…opic operator is removed (#4456)

Backport of #3938 and #4280 to `release-1.6`, together, because #3938
alone breaks every Kafka deletion (#4276) and #4280 is what fixes that.

Kafka suite on this branch fails the delete step on 2 of the last 3 E2E
runs on the container lane (#4328, #4377), both times with
`UninstallFailed ... KafkaTopic/tenant-test/kafka-test-test-results
termination timeout: context deadline exceeded`. Helm removes the topic
operator together with the release, a KafkaTopic that still carries the
`strimzi.io/topic-operator` finalizer then has nobody to release it, so
uninstall waits out its timeout and leaves the topic behind, which is
what #3793 hits on reinstall. When the operator wins the race the same
delete takes 3s. The pre-delete hook from #3938 deletes the release
topics while the operator is still running, #4280 makes it actually
reach the apiserver and reaps its credentials.

All ten commits cherry-picked clean with `-x`. Kafka on this branch is
still ZooKeeper-based, the hook does not depend on that: it selects
topics by `strimzi.io/cluster=<release>`, the label the 1.6 chart puts
on them.

### Testing

- `make unit-tests` green (kafka helm unit tests and
`hack/kafka-pre-delete-hook.bats` included), POSIX sh sweep clean.
- E2E on this PR runs the kafka suite, which now asserts both release
topics are gone after deletion.

```release-note
fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator
```
kubeovn was the last package whose chart and image were produced by a
separate repository: packages/system/kubeovn pulled a tarball from the
latest cozystack/kubeovn-chart tag at update time, and that repository
built and pushed the image. The wrapper existed only to cache the
upstream chart; with the current caching that reason is gone, and the
split cost a second repository, a second release step, and a package
whose provenance a reader could not follow without leaving the tree.

Move the wrapper in, shaped like packages/system/cilium: update pulls
the upstream kube-ovn chart directly and applies patches/mtu.diff, and
image builds images/kubeovn and pins the digest into values.yaml. The
MTU patch already carried monorepo-relative paths, so it applies here
unchanged.

Also carries cozystack/kubeovn-chart#5, which never got a tag: pin
OVN_IPSEC_KEY_DIR under /var/lib. The kube-ovn default sits in
/etc/origin, which a Talos node cannot create, so enabling IPsec left
the node without ESP security associations.

Co-authored-by: Timofei Larkin <[email protected]>
Co-authored-by: Kirill Ilin <[email protected]>
Co-authored-by: Aleksei Sviridkin <[email protected]>
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
(cherry picked from commit 7ebce2e)
Signed-off-by: Myasnikov Daniil <[email protected]>
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.

The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.

Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.

keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
(cherry picked from commit 6cb5793)
Signed-off-by: Myasnikov Daniil <[email protected]>
Signed-off-by: Yan Bondarenko <[email protected]>
(cherry picked from commit 55a2918)
Signed-off-by: Myasnikov Daniil <[email protected]>
Signed-off-by: Yan Bondarenko <[email protected]>
(cherry picked from commit 6e90447)
Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-4253-to-release-1.6 branch from 6e1ecbc to e087e38 Compare September 25, 2026 05:31
@myasnikovdaniil
myasnikovdaniil merged commit 0dc238a into release-1.6 Sep 25, 2026
43 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-4253-to-release-1.6 branch September 25, 2026 09:46
myasnikovdaniil added a commit that referenced this pull request Sep 25, 2026
A hand backport that carries more than one change names all of them in
one phrase -- "Backport of #3938 and #4280", or "Backport of #4253 to
`release-1.6`, together with #3460" for a dependency pulled along --
and the audit read only the first number after "Backport of". The rest
were linked to nothing. A labelled original in second place then read
as MISSING while its backport PR was still open, instead of pending,
and once that PR merged it counted as landed only if the branch history
happened to prove it.

Read the whole reference list the phrase carries, and the together-with
form this repository writes. A list counts in full only when it visibly
ends: at the end of the line or the sentence, or where a "to
release-X.Y" clause names the target line, with the line name a whole
word: release-1.6-fixes, release-1.6.1 and release-1.6.fixes are not
lines, so punctuation after the name ends it only when a space or the
end of the text follows. One that runs on into anything else --
"Backport of #10, #20 is not included", or "#10, #20 to follow in a
separate PR" -- may be saying something about its later items, so only
its first reference, the one the phrase names directly, is kept. The issue a backport fixes, or a CI run it cites further on in
the body, is never taken for an original.

A reference qualified with another repository is now skipped instead of
read as a local number. That also tightens the old first-number rule,
which accepted any owner/repo prefix: "Backport of other/repo#20" linked
local #20, so a merged backport of something unrelated could turn that
PR's MISSING verdict into backported. Only a bare #N or one qualified
with the repository the backport PR itself lives in, compared without
regard to case, is an original.

Over every PR on release-1.4, release-1.5 and release-1.6 this changes
the links of exactly four: #4456 and #4421 each gain #4280, #4377 gains
#4231 and #4328 gains #3460. No verdict moves today, since none of the
added originals carries a backport label.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants