Repository navigation
ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription - #3251
Conversation
The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every guest image pull traverses, containerd and the kernel. Under the peak concurrency of "Install Cozystack" the guests lose cycles to CPU steal, so in-guest readiness/liveness probes miss their deadlines and a HelmRelease intermittently flips InProgress -> Failed with dependents cascading to DependencyNotReady — on a different component each run, the signature of resource thrash rather than a product bug. Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is unchanged (24 vCPU / 72 GiB), so the install does not slow down, while the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation and network overhead the probes were starving against. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughUpdated the e2e job's ChangesCI Runner Update
Estimated code review effort: 1 (Trivial) | ~3 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported. |
## What this PR does The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with three `-drive … if=virtio,format=raw` lines that carry no `cache=` option, so QEMU falls back to its `cache=writeback` default, which honors every guest fsync against the host-backed raw disk files. The management-cluster Talos etcd — which holds every operator's leader-election lease — fsyncs on each Raft commit. During the VM-heavy suites, containerd is simultaneously unpacking dozens of images and CDI is importing multi-GB VM disks onto the same host-backed raw files. The resulting fsync storm serializes etcd's writes into multi-second stalls, so the management apiserver returns `context deadline exceeded` on lease PUTs, and several operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji) lose leadership at the same moment. Whichever chainsaw suite is mid-flight then fails — a different one each run (vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of shared-resource contention rather than a bug in any single suite. This adds `cache=unsafe` to each guest drive. That makes guest flushes no-ops against the host page cache, which removes the fsync-gated etcd stalls at the root of the flakiness. It is safe here precisely because these CI VMs are ephemeral and destroyed at the end of every run — a crash before writeback loses only throwaway state — and it must never be used for real data. The change is minimal and uniform across all three guest drives; no non-guest drive is touched (there are none in this file). Runner sizing alone (#3251) did not resolve this, because more CPU and memory do not change the I/O path: the guest fsyncs still propagate through `cache=writeback` to the host, and etcd still stalls behind the same fsync storm. This addresses the I/O path directly. `git log -S 'cache='` on this file is empty, so a cache mode has never previously been set here. ### Screenshots Not applicable — no UI changes. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the actual diff. The only entries that reference `hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped to node prerequisites — kernel modules, containerd settings, the LVM `global_filter`, and the cluster domain. This change touches none of those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the CI VM `-drive` options, which is invisible to the guest OS and to any real node. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run ``` --- This PR was prepared with the assistance of an AI agent (Claude). A human maintainer reviewed the diff and the reasoning before submission and remains responsible for the change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated end-to-end virtual machine startup testing to use optimized QEMU disk caching settings for improved reliability and performance. * Improved end-to-end Kubernetes cluster readiness checks under load by extending the deadline for reaching “at least 2 nodes Ready,” and aligned related messaging and failure diagnostics accordingly. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR does
The E2E
Install Cozystackgate flakes intermittently: a platform HelmRelease/Deployment sticksInProgress→Failedand its dependents cascade toDependencyNotReady, on a different component each run (observed: opensearch-operator-controller-manager, monitoring-agents-kube-state-metrics, kamaji readiness "context deadline exceeded"). That non-determinism is the signature of host resource thrash, not a product bug.Measured envelope. The sandbox boots three QEMU guests (
hack/e2e-prepare-cluster.bats), each-smp 8 -m 24576:On the 24-vCPU runner the guests alone pin all 24 host vCPUs, leaving nothing for QEMU's own emulation/IO threads, vhost-net, the host
iptables MASQUERADEevery guest image pull traverses, containerd, the kernel and the runner agent. Under the peak concurrency of the platform install the guest vCPUs contend with those host threads and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness/liveness probes miss their deadlines and the HelmRelease flips. Across sampled failed runs the symptom is always a probe/readiness timeout withDependencyNotReady, neverOOMKilled/Evicted/MemoryPressure— CPU steal, not memory, is the binding constraint (RAM already keeps 24 GiB of headroom).Fix. Move the job to the
oracle-vm-32cpu-128gb-x86-64class (already referenced in-repo as the commented alternative). Guest sizing is unchanged (24 vCPU / 72 GiB), so the CPU-bound install does not slow down, while the host keeps 8 vCPU + 56 GiB of headroom that absorbs the emulation/network overhead the probes were starving against.This is deliberately not a retry or timeout bump (forbidden by
docs/agents/e2e-testing.md): the install gate keeps its teeth, runs once, and still fails loud on a genuine product bug. It removes the harness oversubscription that manufactured false failures, and is complementary to the per-component probe-hardening work and to the in-sandbox image-mirror change (which relieves the distinct storage-import axis).Trade-off / caveat. The 32-vCPU class costs more per E2E run and adds load to that runner pool — an infra/cost decision for maintainers. A zero-cost alternative (drop guest
-smpto 6 → 18 vCPU / 6 host headroom on the existing runner) was considered and rejected as primary: it cuts guest compute 25% on an already CPU-bound install and could trade steal-flakes for wall-clock-timeout flakes, unverifiable without a CI run.Screenshots
N/A — CI-only change, no user-facing surface.
Release note
Summary by CodeRabbit