Repository navigation
Move E2E tests out of PR pipeline to release/nightly #2397
Description
Activity
Alternative / complement: path-based Test Impact Analysis on PR
The current proposal is binary — run full E2E nightly/release, nothing on PR. A middle-ground worth considering is Test Impact Analysis (TIA): run only the tests whose code paths a PR actually touches.
Why it fits this repo
The source↔test mapping is already explicit in the layout:
packages/apps/<name>/→hack/e2e-apps/<name>.bats(1:1 for postgres, mariadb, kafka, …)packages/system/*,packages/core/*,hack/*,api/*, root manifests → shared infra → full E2Edocs/,dashboard/, workflow docs → skip E2E
So a rules-based selector on changed paths recovers ~80% of TIA's benefit without coverage instrumentation (which is awkward for bash/bats + Helm anyway).
Selector sketch
PR touches Run only packages/apps/<X>/only <X>.batsonly packages/extra/<X>/only <X>.bats(if exists)only docs/,dashboard/, non-test.github/skip E2E anything in packages/system/*,packages/core/*,hack/,api/, rootfull E2E Trade-offs vs "move to nightly only"
- ✅ Fast feedback preserved for the common case (single-app PRs, which are the majority)
- ✅ Bugs get blamed on the right commit, not the next morning's main run
- ✅ CI hours drop proportionally — if a targeted bats run is ~10 min vs ~75 min full, ~85% savings on single-app PRs
- ✅
/retestcycles shrink because the flakiness surface per PR shrinks - ❌ Adds a path-filter config that needs maintenance as packages evolve
- ❌ Doesn't capture indirect coupling (e.g.
system/monitoringchange that breakspostgres.bats) — needs explicit escape hatches or a conservative fallback-to-full - ❌ Doesn't fix flakiness itself; the stabilization PRs listed above are still needed
Implementation cost
~50 lines: a
dorny/paths-filter(or equivalent) step inpull-requests.yamlthat sets a matrix of bats files to run, with a "run all" fallback when shared infra paths match.Suggestion
Not an alternative to moving full E2E off PR — a complement. Keep the nightly/release full run, and add path-based selection on PR so single-package changes still get targeted verification without the 75 min tax.
References:
- Martin Fowler, The Rise of Test Impact Analysis
- Microsoft Azure Pipelines, Use Test Impact Analysis
- Datadog, How Test Impact Analysis Works
- added and removed
on Jun 4, 2026 Closing — the core objective is shipped and live on
main, and the replacement gate is proven green.Shipped (merged 2026-06-09):
- Single-attempt CI — ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown #2558 dropped the 3× retry on
Run E2E/Install Cozystack. - Diff-aware PR E2E — feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559 landed Test-Impact Analysis (default-on,
full-e2eopt-out), so docs / chart-only / unrelated-app PRs no longer pay for the full suite. - Plus the Stage-1/2 retry-masked bug fixes and harness hardening (fix(installer): bootstrap cozy-system via --create-namespace + label hook #2508, feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) #2509, fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo #2528, fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile #2529, feat(cozyreport): collect Flux/cert-manager/host context + summary.txt #2553–feat(e2e): parallel post-install prep #2557, refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system #2601, feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) #2602, fix(e2e-sandbox): use curl -fsSL when downloading mc #2690, test(e2e): pre-pull platform images via staged-busybox DaemonSet #2724, test(e2e): ouroboros fold + CSI/NFS ordering + OIDC keycloakInternalUrl #2728, fix(platform): order cozystack-basics after the APIs its admission policies type-check #2842, feat(api): expose HelmRelease generation knobs as cozystack-api flags #2571).
Full-suite coverage now runs on the release-cut PR — confirmed green. The v1.5.0 cut (#2970) ran
E2E Teststo success on the digest-bake commit (e1aaa90e), which touchespackages/core/and so escalates to the full suite under TIA. The standalonerelease-e2e.yamltag run on that same commit was red — a workflow-level issue, not a product regression — so it's a redundant, non-blocking alarm, being retired in #2978.Spun-off follow-ups: Chainsaw migration → #2979, per-test cleanup into bats
teardown()→ #2583.Not built here: a scheduled nightly full-suite run against
mainbetween releases (coverage currently lands at release-cut time). If we want periodic main coverage, let's open a dedicated issue.- Single-attempt CI — ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown #2558 dropped the 3× retry on
- added a commit that references this issue
on Jun 25, 2026 - added a commit that references this issue
on Jun 25, 2026
Metadata
Metadata
Labels
Type
Projects
- Status
Problem
E2E tests on every PR are expensive and provide low signal:
Current state (2026-06-08)
Consolidated test PR #2500 is now closed (2026-05-14) — every piece shipped via its own focused PR, so the preview snapshot is no longer needed.
Program is ~85% complete: every retry-masked bug fix and test-suite improvement has shipped, and the install gate has been hardened so single-attempt CI will actually fail on a broken release. The two keystone PRs remain open: #2558 (drop-retries) was rebased and re-ran CI today (2026-06-08) — it now also waits out the gateway.bats tenant teardown — but is still
REVIEW_REQUIRED; #2559 (TIA + release E2E workflow) is unchanged and still awaiting review. Separately, #2826 opens a new exploration: migrating the app E2E suite from BATS to declarative Chainsaw.Shipped — bugs the 3× retry was hiding
Audit of 30 successful PR runs found deterministic bugs that retry was masking. Each got a focused fix:
installer: drop chart-defined Namespace; caller pre-applies (Helm namespace-ownership conflict on cold install, 17/17 hit rate)operator: 5 HR generation knobs +Strategy=RetryOnFailure,RetryInterval=30s(replaces theflux reconcile --forceworkaround for seaweedfs-system HR 2-min timeout)harbor: bucket-secret values-driven; HR gated onBucketInfovia FluxvaluesFromobjectstorage-controller: retry on Bucket update conflict during BucketAccess reconcileShipped — test-suite improvements
cozyreport: collect Flux / cert-manager / host context +summary.txt(machine-readable "what's broken now")e2e: replace fixed timeouts with event-driven backstops across 16 bats filesvirtual-machine: vminstance disk delete-recreate race + bump VM IP/ready timeoutse2e: pre-pull timing-sensitive platform images before installinstaller,e2e: configurablehelmReleaseInterval+ parallel post-install prep (LINSTOR / SC / MetalLB runs alongside platform reconcile wait, eliminates the ~10-min Flux dead zone inInstall Cozystack)Shipped — additional E2E hardening (post 2026-05-13)
Continuation of the program scope after the May-13 status update:
fix(e2e-sandbox): usecurl -fsSLwhen downloadingmc(fail fast on a bad download instead of silently producing a broken image)test(e2e): pre-pull platform images via staged-busybox DaemonSet (DaemonSet-based extension of #2556; pre-warms the pull on every node in parallel before install)test(e2e): ouroboros fold + CSI/NFS ordering + OIDCkeycloakInternalUrlShipped — additional E2E hardening (post 2026-05-28)
test(e2e): fail the install test when HelmReleases are not Ready — the backgroundedkubectl waitfan-out discarded child exit codes and the final check never exited non-zero, so the readiness gate was toothless. Prerequisite for meaningful single-attempt CI (#2558): the gate has to actually fail before retries can be dropped.67b1a960d; rebase-merged, GitHub shows the PR closed)refactor(seaweedfs): splitseaweedfs-systemintoseaweedfs-db+seaweedfs-system— retires the slow-HR install path that #2509'sRetryOnFailureknob was working aroundfeat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds the #2612 kubevirt TPM fix) — reduces install-time reconcile flakeOpen — keystone PRs
ci: drop 3× retry onRun E2E+Install Cozystack(keeps the 3× retry only onPrepare environment, pure infra). On failure now captureskubectl get hr -A -o wide+ events. Keystone — relies on all the bug-fixes above to keep CI green without retries.REVIEW_REQUIRED. Rebased + re-ran E2E 2026-06-08; now also waits out the gateway.bats tenant teardownci: TIA E2E (default-on,full-e2eopt-out) + release E2E workflow. This is the direct implementation of this issue's original proposal: graph-aware test selection on PRs + a separate release-branch E2E run.REVIEW_REQUIRED(no change since 2026-05-28)Open — additional E2E tooling
feat(hack): improvecheck-readiness.sh— wait/core/parallel modes and broader coveragefeat(hack): rewritecheck-readinessas a Go command with test suite (supersedes #2740 once landed)Exploration — declarative E2E (Chainsaw)
test(e2e): pilot migrating app E2E from BATS to Kyverno Chainsaw (CNCF, KUTTL successor). Portspostgres+bucketas examples underhack/e2e-chainsaw/, duplicating (not replacing) the BATS originals. Declarative tests are easier to select per-PR (complements the TIA work in #2559) and remove a class of bash-level flakiness.In-flight — deterministic bug E2E surfaced
fix(platform): ordercozystack-basicsafter the APIs its ValidatingAdmissionPolicies type-check — kube-controller-manager's VAP status type-checker nil-derefs when a policy matches a CRD whose schema isn't established yet, CrashLooping KCM on a cold install. Exactly the kind of deterministic, ordering-dependent failure that retries used to paper over.Followups
feat(api): bringcozystack-api'sconvertApplicationToHelmReleaseto HR-knob parity with the operator (followup to feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) #2509). OPEN (currently conflicting — needs rebase).chore: address review followups from feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) #2509 (operator min-duration validator) + feat(cozyreport): collect Flux/cert-manager/host context + summary.txt #2553 (cozyreport pipefail / OOMKilled lastState). OPEN, draft.e2e: move per-test resource cleanup into bats teardown(). Carved out from review of fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts #2555. OPEN issue.Expected impact (revisited)
Related: prior E2E stabilization efforts
[tests] Fix flaky E2E timeouts, add diagnostics and retry cleanup(Andrei Kvapil (@kvaps), open)[tests] Stabilize E2E kubernetes tests(Aleksei Sviridkin (@lexfrei), merged)[tests] Fix Kafka E2E test timeout and retry race condition(Aleksei Sviridkin (@lexfrei), merged)fix(e2e): increase HelmRelease readiness timeout for kubernetes test(Aleksei Sviridkin (@lexfrei), merged)fix(e2e): make kubernetes test retries effective by cleaning up stale resources(Aleksei Sviridkin (@lexfrei), merged)[e2e] fix timeouts for capi and keycloak(Andrei Kvapil (@kvaps), merged)Data (original baseline, Dec 2025 – Apr 2026)
Analysis based on 773 E2E runs from Dec 2025 – Apr 2026: