Skip to content

ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription - #3251

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
ci/e2e-runner-cpu-headroom
Jul 8, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
ci/e2e-runner-cpu-headroom

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

The E2E Install Cozystack gate flakes intermittently: a platform HelmRelease/Deployment sticks InProgress → Failed and its dependents cascade to DependencyNotReady, on a different component each run (observed: opensearch-operator-controller-manager, monitoring-agents-kube-state-metrics, kamaji readiness "context deadline exceeded"). That non-determinism is the signature of host resource thrash, not a product bug.

Measured envelope. The sandbox boots three QEMU guests (hack/e2e-prepare-cluster.bats), each -smp 8 -m 24576:

Resource Guests demand 24-vCPU runner (current) 32-vCPU runner (this PR)
vCPU 3 × 8 = 24 24 → 0 host headroom 32 → 8 (25%) headroom
RAM 3 × 24 = 72 GiB 96 GiB → 24 GiB headroom 128 GiB → 56 GiB headroom

On the 24-vCPU runner the guests alone pin all 24 host vCPUs, leaving nothing for QEMU's own emulation/IO threads, vhost-net, the host iptables MASQUERADE every guest image pull traverses, containerd, the kernel and the runner agent. Under the peak concurrency of the platform install the guest vCPUs contend with those host threads and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness/liveness probes miss their deadlines and the HelmRelease flips. Across sampled failed runs the symptom is always a probe/readiness timeout with DependencyNotReady, never OOMKilled/Evicted/MemoryPressure — CPU steal, not memory, is the binding constraint (RAM already keeps 24 GiB of headroom).

Fix. Move the job to the oracle-vm-32cpu-128gb-x86-64 class (already referenced in-repo as the commented alternative). Guest sizing is unchanged (24 vCPU / 72 GiB), so the CPU-bound install does not slow down, while the host keeps 8 vCPU + 56 GiB of headroom that absorbs the emulation/network overhead the probes were starving against.

This is deliberately not a retry or timeout bump (forbidden by docs/agents/e2e-testing.md): the install gate keeps its teeth, runs once, and still fails loud on a genuine product bug. It removes the harness oversubscription that manufactured false failures, and is complementary to the per-component probe-hardening work and to the in-sandbox image-mirror change (which relieves the distinct storage-import axis).

Trade-off / caveat. The 32-vCPU class costs more per E2E run and adds load to that runner pool — an infra/cost decision for maintainers. A zero-cost alternative (drop guest -smp to 6 → 18 vCPU / 6 host headroom on the existing runner) was considered and rejected as primary: it cuts guest compute 25% on an already CPU-bound install and could trade steal-flakes for wall-clock-timeout flakes, unverifiable without a CI run.

Screenshots

N/A — CI-only change, no user-facing surface.

Release note

NONE

Summary by CodeRabbit

  • Chores
    • Increased the default runner size for end-to-end pull request checks.
    • Added clearer workflow guidance to reduce flaky readiness and liveness failures during installation steps.

The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU
in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host
headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every
guest image pull traverses, containerd and the kernel. Under the peak
concurrency of "Install Cozystack" the guests lose cycles to CPU steal,
so in-guest readiness/liveness probes miss their deadlines and a
HelmRelease intermittently flips InProgress -> Failed with dependents
cascading to DependencyNotReady — on a different component each run, the
signature of resource thrash rather than a product bug.

Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is
unchanged (24 vCPU / 72 GiB), so the install does not slow down, while
the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation
and network overhead the probes were starving against.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@coderabbitai

coderabbitai Bot commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: aba7e421-c8ce-416f-b7d3-958fc31477bd

📥 Commits

Reviewing files that changed from the base of the PR and between 6c69c00 and 08ac063.

📒 Files selected for processing (1)
  • .github/workflows/pull-requests.yaml

📝 Walkthrough

Walkthrough

Updated the e2e job's runs-on runner specification in the pull-requests GitHub Actions workflow from a 24 vCPU/96 GiB runner to a 32 vCPU/128 GiB runner, and added an extended comment documenting resource contention issues affecting QEMU guest probes.

Changes

CI Runner Update

Layer / File(s) Summary
Increase e2e runner size and document rationale
.github/workflows/pull-requests.yaml
Changes non-debug runs-on from 24 vCPU/96 GiB to 32 vCPU/128 GiB and replaces prior comment with a detailed explanation of guest/host CPU contention causing flaky readiness/liveness checks.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Suggested labels: kind/bug

Suggested reviewers: lllamnyp, androndo, kvaps

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main CI change: moving the E2E install gate to a larger 32-vCPU runner to avoid oversubscription.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/e2e-runner-cpu-headroom

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/S This PR changes 10-29 lines, ignoring generated files area/testing Issues or PRs related to testing (e2e, bats, unit tests) labels Jul 8, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review July 8, 2026 22:20
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

@dosubot dosubot Bot added the area/ci Issues or PRs related to CI workflows, GitHub Actions, automation label Jul 8, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) merged commit a53d64d into main Jul 8, 2026
43 of 45 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the ci/e2e-runner-cpu-headroom branch July 8, 2026 22:49
Andrei Kvapil (kvaps) added a commit that referenced this pull request Jul 29, 2026
## What this PR does

The e2e QEMU guests are launched in `hack/e2e-prepare-cluster.bats` with
three `-drive … if=virtio,format=raw` lines that carry no `cache=`
option, so QEMU falls back to its `cache=writeback` default, which
honors every guest fsync against the host-backed raw disk files.

The management-cluster Talos etcd — which holds every operator's
leader-election lease — fsyncs on each Raft commit. During the VM-heavy
suites, containerd is simultaneously unpacking dozens of images and CDI
is importing multi-GB VM disks onto the same host-backed raw files. The
resulting fsync storm serializes etcd's writes into multi-second stalls,
so the management apiserver returns `context deadline exceeded` on lease
PUTs, and several operators (capk, etcd-operator, fdb-operator,
cilium-operator, kamaji) lose leadership at the same moment. Whichever
chainsaw suite is mid-flight then fails — a different one each run
(vmcluster, kubernetes-latest, vmdisk), which is the fingerprint of
shared-resource contention rather than a bug in any single suite.

This adds `cache=unsafe` to each guest drive. That makes guest flushes
no-ops against the host page cache, which removes the fsync-gated etcd
stalls at the root of the flakiness. It is safe here precisely because
these CI VMs are ephemeral and destroyed at the end of every run — a
crash before writeback loses only throwaway state — and it must never be
used for real data. The change is minimal and uniform across all three
guest drives; no non-guest drive is touched (there are none in this
file).

Runner sizing alone (#3251) did not resolve this, because more CPU and
memory do not change the I/O path: the guest fsyncs still propagate
through `cache=writeback` to the host, and etcd still stalls behind the
same fsync storm. This addresses the I/O path directly.

`git log -S 'cache='` on this file is empty, so a cache mode has never
previously been set here.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
actual diff. The only entries that reference
`hack/e2e-prepare-cluster.bats` (ansible-cozystack and ccp) are scoped
to node prerequisites — kernel modules, containerd settings, the LVM
`global_filter`, and the cluster domain. This change touches none of
those: it adds a QEMU host-side hypervisor flag (`cache=unsafe`) to the
CI VM `-drive` options, which is invisible to the guest OS and to any
real node. No downstream repository is affected.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): boot e2e QEMU guests with cache=unsafe so ephemeral CI VM fsyncs no longer stall the management-cluster etcd, removing the cross-operator leader-election flakiness that failed a different chainsaw suite each run
```

---

This PR was prepared with the assistance of an AI agent (Claude). A
human maintainer reviewed the diff and the reasoning before submission
and remains responsible for the change.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Updated end-to-end virtual machine startup testing to use optimized
QEMU disk caching settings for improved reliability and performance.
* Improved end-to-end Kubernetes cluster readiness checks under load by
extending the deadline for reaching “at least 2 nodes Ready,” and
aligned related messaging and failure diagnostics accordingly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Issues or PRs related to CI workflows, GitHub Actions, automation area/testing Issues or PRs related to testing (e2e, bats, unit tests) size/S This PR changes 10-29 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants