Skip to content

[Backport release-1.6] fix(e2e): disable guest fsync on ephemeral CI VM disks - #4431

Merged
myasnikovdaniil merged 2 commits into
release-1.6from
backport-3455-to-release-1.6
Sep 24, 2026
Merged

myasnikovdaniil merged 2 commits into
release-1.6from
backport-3455-to-release-1.6

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Backport of #3455 to release-1.6.

Management etcd on this branch stalls under the VM-heavy suites the same way main did before #3455: every operator loses its lease in the same few seconds, and install failed on it in #3550 and #3905. Second commit widens tenant node-join deadline from 12m to 18m. Most kubernetes suite failures on 1.6 are workers that get their CSR signed 6-9 minutes after VMI start and miss the 12m window, the script step already has 40m so the wider wait fits.

Both commits cherry-picked clean with -x.

Testing

  • make unit-tests green, POSIX sh sweep clean.
  • Real check is E2E on this PR.
NONE

The e2e QEMU guests boot with three raw virtio -drive lines that carry
no cache= option, so QEMU defaults to cache=writeback and honors every
guest fsync. The management-cluster Talos etcd fsyncs on each Raft
commit, while containerd unpacks dozens of images and CDI imports
multi-GB VM disks onto the same host-backed raw files during the
VM-heavy suites. The fsync storm serializes etcd's writes into
multi-second stalls, the mgmt apiserver times out lease PUTs, and many
operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji)
lose leadership at once — failing whichever chainsaw suite is
mid-flight, a different one each run.

Add cache=unsafe to each guest drive so guest flushes become no-ops
against the host page cache, removing the fsync-gated etcd stalls. This
is safe only because these CI VMs are ephemeral and destroyed every
run; never use it for real data.

Assisted-By: Claude <[email protected]>
Signed-off-by: Andrei Kvapil <[email protected]>
(cherry picked from commit ad35afa)
Signed-off-by: Myasnikov Daniil <[email protected]>
The kubernetes-latest and kubernetes-previous suites flaked on the
worker node-join wait under host load. In run 30260770694 both failed at
exactly 12m with zero tenant Nodes registered and the tenant cilium HR
still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed
the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m
into this wait and the guests then had ~10m to import Talos, boot,
register a kubelet and let cilium turn the nodes Ready.

This single deadline absorbs the entire worker bring-up (the
machinedeployment replicas=2 gate clears before the VMIs even exist), so
under storage pressure 12m is too tight. A less-loaded fleet run passed
both suites unchanged, confirming load-induced slowness rather than a
stuck bring-up. Widen the deadline to 18m, which restores margin and
still sits well inside the 40m chainsaw step timeout.

Assisted-By: Claude <[email protected]>
Signed-off-by: Andrei Kvapil <[email protected]>
(cherry picked from commit 330acfe)
Signed-off-by: Myasnikov Daniil <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: f0530dd3-5520-488b-9a04-c79399d9c5b4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug labels Sep 24, 2026
@myasnikovdaniil
myasnikovdaniil merged commit d6c44d5 into release-1.6 Sep 24, 2026
13 of 14 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3455-to-release-1.6 branch September 24, 2026 09:26
myasnikovdaniil added a commit that referenced this pull request Sep 25, 2026
The audit started from labels and dropped every backport PR whose
original was not a candidate for the branch. On release-1.6 that hid
nine hand backports of unlabelled main PRs (#4431 to #4435, #4437,
#4438, #4456 and #4475), and it had no way to notice two open backport
PRs for the same originals: #4421 sat open next to #4456 for #3938 and
#4280, and nothing reported it.

Read the PRs on the branch from the side of the originals they claim as
well, and report two more sections per branch.

UNLABELLED lists each original claimed by backport PRs on the branch
that is not a candidate for it, with every backport PR claiming it and
its state, and says whether they backported it, only claim it with a PR
still open, or were all closed. It never moves the exit code: the gate
answers whether everything labelled landed, and an unlabelled backport
can only add to a branch, never leave a labelled change off it.

DUPLICATE lists each original claimed by two open backport PRs, or by
an open one after another already merged, labelled or not. Both make
the audit exit 1. One of the PRs is redundant, or the open one is the
rest of a split backport; either way someone has to decide before the
cut, and no verdict can say so, since a verdict settles on the first
merged backport or reports the first open one as pending. A closed PR
next to an open one is how a conflicting bot backport gets redone by
hand and is not flagged.

The titles of originals that no listing carries come from one GraphQL
request for the whole run. A failed lookup costs the titles and nothing
else: the URL is derived locally and the exit code is already settled.

--json now emits an object per branch holding candidates, unlabelled
and duplicates arrays, in place of the bare array of verdicts.

On release-1.6 today this lists 13 unlabelled originals, the nine above
among them, and three duplicates that are open right now: the bot's
conflict drafts for #3936, #4254 and #4292 were left open next to their
hand backports, and #4254 and #4292 each also have a fork PR open next
to its reopening from a branch in this repository. #4421 is closed and
shows up only as a closed claim on #4280.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants