Repository navigation
ci(e2e): run nightly and release-tag e2e on the container lane - #4828
Merged
Merged
Conversation
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The comment on every lane claimed the budget fits two tenant Kubernetes suites failing at their node-join deadline. A green full suite on the container lane already spends most of it, and a suite that runs into the deadline costs about 25 minutes more than one that passes, so the claim does not hold. State the real bound so a cancelled run is read as a red suite, not as a budget to raise. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
requested a review
from Andrei Kvapil (kvaps)
as a code owner
October 9, 2026 16:25
Aleksei Sviridkin (lexfrei)
added
the
kind/backport
Aleksei Sviridkin (lexfrei)
requested review from
IvanHunters,
Timofei Larkin (lllamnyp),
myasnikovdaniil and
Timur Tukaev (tym83)
as code owners
October 9, 2026 16:25
github-actions
Bot
added
area/testing
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (11)
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughNightly and tagged E2E jobs now use Talos containers instead of booting the nocloud disk. The workflows prepare and remove container clusters, use local storage with DRBD disabled, and update checks and documentation to describe the resulting lane coverage. ChangesE2E lane transition
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant NightlyE2EWorkflow
participant GitHubRunner
participant GHCR
participant ContainerCluster
participant E2ESuite
NightlyE2EWorkflow->>GitHubRunner: Start E2E after mirror
GitHubRunner->>GHCR: Log in and stage the published closure
GitHubRunner->>ContainerCluster: Load modules and run prepare-env-container
GitHubRunner->>ContainerCluster: Install with DRBD disabled and local storage
GitHubRunner->>E2ESuite: Run the full E2E suite
GitHubRunner->>ContainerCluster: Delete the container cluster
Merge Risk: ⚪ Minimal · up to The container-lane transition is mergeable after normal workflow checks. The loss of disk- and DRBD-specific CI coverage is documented and intentional. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
IvanHunters
approved these changes
Oct 9, 2026
Aleksei Sviridkin (lexfrei)
enabled auto-merge
October 9, 2026 16:26
Contributor
Author
|
The |
|
Git push to origin failed for release-1.6 with exitcode 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Nightly and the release-tag e2e (
e2e-tag.yaml, the rc gate) now run on the Talos-container substrate the PR lanes moved to in #4020. This is the interim step from #4642, not the whole issue, so it does not close it.On QEMU the nightly full suite has been cancelled at its 215-minute timeout on every night since 2026-10-01 except 2026-10-07. From 2026-10-02 on, kubernetes-latest and kubernetes-previous failed their tenant node-join deadline almost every night, each failure took about 35 minutes, and the suites after them alphabetically never ran. In the 18 PR runs since 2026-10-03 that I checked and that ran them, they passed every time, at 7 to 10 minutes each. A nightly dispatched from this branch (run 37934155231) passed all 62 tests in 193 minutes. The kubernetes suites took 9 to 11 minutes there.
Changes:
nightly.yamlande2e-tag.yamlget the same host module step,prepare-env-container, endpoint and storage-class env and two-part teardown aspull-requests.yaml. Nightly's e2e no longer waits for the nocloud disk, which it does not boot. The disk is still built and published.e2e-tag.yamlstill requires the release to carry its disk but no longer downloads it. A dispatch against a tag cut before the container lane (v1.6.3 and older, v1.7.0-alpha.1) fails early and says to dispatch with--ref <tag>. Whentags.yamlcalls it, it runs the copy from the tag's own commit, so the guard does not fire there.timeout-minutes: 215in all four e2e workflows claimed the budget covers two failing tenant Kubernetes suites. It does not, and the comment now says so. The value is unchanged.platform-packages_test.batsnow checks the container-lane env on all four workflows, andcollect-images_test.batscovers the new guard.What loses coverage: DRBD and the
replicatedStorageClass (Immediate binding, tenant StorageClass propagation, RWX), the Cozystack Talos node image with its extensions, and the stock 600M CDI importer limit. None of them can run on containers, so no CI lane tests them until #4642 gets a lane with a kernel per node. The docs and comments that pointed at nightly and rc as that coverage now say this.Release lines keep their own
e2e-tag.yaml, so rc tags on release-1.6 stay on QEMU until this is backported. That is whykind/backportis on this PR. Leftover QEMU-only text and diagnostics are collected in #4825.A dispatch of
e2e-tag.yamlfrom this branch againstv1.7.0-alpha.3is running as run 37958852914. I'll post the result here.Screenshots
Downstream repositories
Release note
Summary by CodeRabbit