Repository navigation
e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642
Description
PR e2e runs srv1-srv3 as talos containers since #4020 (backported to release-1.6 in #4437), but nightly.yaml and e2e-tag.yaml still boot three QEMU guests with make prepare-env. tags.yaml calls e2e-tag.yaml as the rc e2e gate, so release candidates run on QEMU too. Tenant worker sits at L3 there, and suites that pass on containers fail on QEMU.
What is red
rc e2e failed on v1.6.2-rc.1, v1.6.3-rc.1, v1.6.3-rc.2 and v1.6.4-rc.1. Failed suites are known for two of them: kubernetes-previous and kubernetes-latest on v1.6.2-rc.1, the same two plus mariadb-single on v1.6.4-rc.1. The two kubernetes suites ran 1292-2364s on QEMU in the runs listed here, against 466-809s on containers (numbers from #4020).
Nightly e2e failed every night from 2026-09-04 to 2026-09-14 and again on 2026-09-30, last green one was 2026-09-03. Failed suites on 09-14 and 09-30 are opensearch and both kubernetes ones. Nights in between stopped on the mirror job, that is #4423 and not part of this.
#4617 puts the kubernetes failures on 09-30 on replicated cleanup that wants 90 GiB free per node, with the kafka suite leaving about 40 GiB behind (#4616).
Last three stable releases, v1.6.2, v1.6.3 and v1.6.4, were promoted with skip_e2e_gate: true (v1.6.2, v1.6.3, v1.6.4). So rc e2e blocks nothing now.
What only QEMU covers
Container lane has one kernel for all three nodes, so no DRBD (details in #4020). These left the PR path and were supposed to stay covered nightly and on rc, where they are red or bypassed now:
- volume on
replicatedStorageClass survives stopping one node,Immediatebinding, linstor withdrbd.enabled - cozystack talos image with
drbd,zfsandopenvswitchextensions (container lane boots upstream siderolabs image) - tenant StorageClass propagation for remote classes and RWX NFS through
kubevirt-csi, inhack/e2e-chainsaw/_lib/run-kubernetes.sh, runs only withreplicated - tenant kubernetes on replicated worker disks,
kubernetes-latestandkubernetes-previous
Two more are not tested by any lane, no e2e suite on main drains a node or migrates a VMI: live migration of a VMI with replicated RWX disk (third item of #4002), and host eviction with live migration of a tenant worker (#4291).
Proposal
Container lane stays the only lane for PRs. I see no point in a third environment next to it, QEMU is half-way: tenant VMs nested three levels, CNCF runners can't run that reliably, and it goes red on suites that are green on containers.
Everything from the list above should run on a temporary three-node cluster created for the run and deleted after, so nodes have own kernels and tenant VMs sit at L2. It needs design first: provisioning from CI job in OCI, quota, cost, when it runs (rc, and PRs touching linstor, piraeus-operator, kubevirt, kubevirt-csi or cluster-api providers). We are not ready for this now.
Until then, run nightly and rc e2e on container lane, same swap #4020 did in the PR jobs (prepare-env-container, COZY_E2E_STORAGE_CLASS: local). rc gate checks something again and skip_e2e_gate goes back to emergency use. Price is that cozystack talos image and DRBD are not booted on management nodes anywhere in CI until the cluster exists, so the list above stays untested. This replaces #4002.
Open
- can a VMI live migrate with software emulation on container lane? Not checked, if yes live migration test moves there
- tenant StorageClass propagation can be checked by rendering the kubernetes chart against synthetic remote linstor classes, like
verify_storageclass_fallback_defaultinrun-kubernetes.shdoes. Not checked whether tenantcsirelease gets Ready with no workers e2e-tag.yamlon containers against a published tag not tried
Activity
lexfrei commentedon Oct 9, 2026
The nightly e2e job now gets cancelled instead of failing: it hits
timeout-minutes: 215every night since 2026-10-01 except 2026-10-07, with "The job has exceeded the maximum execution time of 3h35m0s" in the annotations.The time goes into tenant node-join on this lane. From 2026-10-02 on, kubernetes-latest and kubernetes-previous fail every night with "fewer than 2 tenant nodes Ready within 29m", and on 10-08 and 10-09 computeplane fails as well. A failing suite runs 34 to 39 minutes, a passing one 13 to 17. That is about an hour extra, so the job is killed before the end of the alphabetical suite order. On 10-09 it died inside postgres-2-backup-roundtrip, which means qdrant, rabbitmq, redis, seaweedfs, securitygroup, valkey, vmdisk and vminstance got no nightly run at all. Cancelled runs also upload no chainsaw report.
In those runs both worker VMIs are Running, while the tenant cilium HelmRelease fails on
Deployment/cozy-cilium/cilium-operator status: 'Failed'. I didn't go further than that. The comment abovetimeout-minutes: 215innightly.yamlsays the budget covers a failing tenant-Kubernetes pair, which is no longer true. Even the green 10-07 run used 207 of the 215 minutes.ci(e2e): run nightly and release-tag e2e on the container lane (#4828)