[Tracking] Remaining CI / build / release / E2E automation work #3278

Summary

Umbrella tracking issue for the remaining CI / build / release / E2E-testing automation work. It is a companion to the individual epics (#2937, #2677, #2499, #2847, #2653, #2585, #3238, #4642) and gives one place to see what is done, what is ready, and what is externally blocked. Action IDs (A1-A19) are stable references.

Update 2026-10-01

The E2E lane items were re-checked first. The remaining actions were checked against main at bc400dff1 on 2026-10-02 and the bullets below carry the corrections, the 2026-08-13 text further down is the older snapshot.

A14 and A15 are done via #4020 (2026-09-07, backported in #4437): srv1-srv3 of both merge-gating lanes run as Talos containers, and #3238 is closed. What the container lane cannot run (DRBD, the replicated StorageClass, live migration, the cozystack talos image) is the new action A18, tracked in #4642. nightly.yaml and e2e-tag.yaml still boot QEMU, the rc e2e job failed on v1.6.2-rc.1, v1.6.3-rc.1, v1.6.3-rc.2 and v1.6.4-rc.1, and v1.6.2, v1.6.3 and v1.6.4 were promoted with skip_e2e_gate: true. The nightly mirror job failure since 2026-09-15 is #4423 and is not part of A18.

Nightly is worse than the 2026-08-13 text says. The last scheduled green was 2026-09-03, before that 2026-08-17, and Artifact Retention has been red on every scheduled run since 2026-08-18 (new action A19). Promote RC skipped the e2e gate on all five runs since 2026-08-05 (v1.6.1, v1.6.2, v1.5.4, v1.6.3 and v1.6.4), each logged Promoting <rc> without verified green full E2E evidence.

Status as of 2026-08-13

Re-verified against main at 09dc7f47f plus live PR, run and registry state. Two actions moved to done since the 2026-07-20 snapshot: A5, because promote-rc.yaml has now cut two releases live (v1.6.0 on 2026-07-22 and v1.6.1 on 2026-08-05), and A16, closed by #3441 on 2026-08-10, which removes the sequencing constraint it placed on A9. One diagnosis is obsolete rather than resolved. A4's "nightly has never completed a run" was fixed by #3332 and nightly published for ten days, but it has failed every scheduled run since 2026-08-02 on its E2E (full suite) job. That failure is known and is being worked, so what changes here is the recorded cause, not the ownership. A17 is now partial, the specific retired-digest blocker is gone and the systemic preflight is not. The other twelve actions are unchanged, and none has been overtaken by the Chainsaw migration.

One sequencing change: A9 is gated on measurement rather than merely unblocked. Its payoff rests on a "~60% of PRs touch one app" figure that has never been measured, while docs/agents/e2e-testing.md §7 records that most PRs legitimately escalate to the full suite. The install-telemetry task in #2847 Track A is A9's prerequisite. The escalation surface that measurement would run against moved three times in the six days to 2026-08-10 and settled when #3434 closed on 2026-08-13, so a number taken now describes post-#3441 behaviour and should record the commit it was taken at.

Status snapshot

Done since the 2026-07-20 snapshot

• A5, the promote flow is proven live, twice. promote-rc.yaml succeeded on 2026-07-21, twice on 2026-07-22 and on 2026-08-05, covering v1.6.0-rc.4 → v1.6.0 (2026-07-22) and v1.6.1-rc.1 → v1.6.1 (2026-08-05). cut-prerelease.yaml succeeded four times of five. Update 2026-10-02: the promotes since then (v1.6.2, v1.5.4, v1.6.3, v1.6.4) succeeded too, all with the e2e gate skipped, see A18.
• A16, fix(ci): decide E2E test coverage per changed path #3441 (2026-08-10) closed select-e2e.sh: a per-suite test edit silently defeats the full-suite escalation #3330 and ci: select-e2e maps only *-application sources, so kuberture and securitygroup suites are never selected by dependency #3665. Coverage is now decided per changed path instead of inferred from an empty merged selection, so a per-suite test edit can no longer swallow a system-package escalation, see the "Coverage is decided per changed path" bullet in docs/agents/e2e-testing.md §7. Related: fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep #3670 (2026-08-08) closed e2e: an empty TIA selection concludes "E2E Tests" green with no suite run #3392, so an unclassified path escalates rather than producing an empty selection that both lanes read as "skip Chainsaw" before posting E2E Tests green (hack/select-e2e.sh:322). A9 no longer has a correctness prerequisite ahead of it, what stands in front of it now is the measurement described under "Ready now".
• A14 and A15, test(e2e): run the merge-gating lanes on Talos containers #4020 (2026-09-07, backport [Backport release-1.6] test(e2e): run the merge-gating lanes on Talos containers #4437 on 2026-09-24) runs srv1-srv3 of both merge-gating lanes as Talos containers on the LINSTOR local StorageClass. test(e2e): lightweight E2E cluster (kind / Talos-in-Docker) for diff-scoped PR runs #3238 was closed on 2026-10-01 as done via test(e2e): run the merge-gating lanes on Talos containers #4020. The suites and cases that still need a real cluster are listed in e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642 (A18). The OpenEBS LocalPV hostpath package from feat(local-storage): add opt-in OpenEBS LocalPV hostpath package #3266 (open, changes requested) is not part of it, the lane runs on the LINSTOR local StorageClass.
• A13, namespaced backport labels (chore(ci): align backport labels with namespaced label convention #2585, closed 2026-08-27). The adjacent fix did not land, full-e2e is still not declared in .github/labels.yml and survives only because the label sync runs with delete-other-labels: false.

Carried forward from earlier snapshots, still true:

• A8, diff-driven install selector merged (test(e2e): diff-driven install selector select-install.sh #3279, 2026-07-15). hack/select-install.sh + hack/select-install_test.bats are on main and run on every PR via Makefile:161 → make unit-tests → pull-requests.yaml:212. Sub-epic [Epic] Diff-driven E2E install selector — hack/select-install.sh + test-minimal (#2847 Track A) #3277 closed as completed. Note the selector is still not invoked by any workflow, so its closure has never been exercised against a real install, and that is A9's job.
• A10, Chainsaw app-suite migration merged (test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826, 2026-07-12): 22 suite directories under hack/e2e-chainsaw/, 21 runnable, backup parked as chainsaw-test.yaml.disabled. The migration issue test(e2e): migrate the E2E app suite from BATS to Kyverno Chainsaw #2979 was closed on 2026-08-14 as done. The two unported files, hack/e2e-apps/monitoring-oidc-customconfig.bats and hack/e2e-apps/monitoring-oidc-system.bats, moved to [Tracking] Shell test architecture: vanilla BATS for unit tests, assert library, testable suite shell #3453 item 7, where the point is that no lane invokes either of them, so monitoring OIDC has e2e coverage on paper and none in practice. Update 2026-10-02: 23 suites are runnable now, backup and kubernetes-proxmox are parked, and the two monitoring OIDC BATS suites were deleted in test(e2e): remove unused monitoring suites and audit BATS runners #4019 (2026-09-25), which closed [Tracking] Shell test architecture: vanilla BATS for unit tests, assert library, testable suite shell #3453 item 7.
• A1 (code), every workflow moved off the persistent self-hosted runner incl. the debug reroute (ci: move remaining workflows off the persistent self-hosted runner #3268, 2026-07-09). Every runs-on on main is ubuntu-* or oracle-vm-*, except e2e-proxmox.yaml, which uses [self-hosted, proxmox] on purpose and is gated behind the PROXMOX_NIGHTLY variable (unset). Residual is the physical host retirement plus rehoming the breakpoint rendezvous server, which is still load-bearing: the breakpoint step reaches it through the BREAKPOINT_ENDPOINT repository variable (pull-requests.yaml:1026).

Corrections to the previous snapshot

• A4's diagnosis is obsolete, the symptom is not. fix(ci): resolve nightly packages digest without image-specific skopeo ops #3332 merged 2026-07-20 and it worked: nightly completed successfully on 2026-07-22, 07-23, 07-24, 07-30, 07-31 and 08-01, and both artifacts exist today, an anonymous GHCR manifest GET returns 200 for cozy-installer:nightly and 200 for cozystack-packages:nightly, where both previously 404'd. What is true now is a different failure: 12 consecutive scheduled failures since 2026-08-02, and on the latest run the failing job is E2E (full suite) at Run E2E tests (full suite) while Mirror images to GHCR and Build & publish nocloud disk both succeed. Six of the last 29 scheduled runs passed, plus one manual dispatch on 2026-07-21. The current failure is known and is actively being worked, so A4's remaining repo-side content is the RETENTION_APPLY flip, and what this correction changes is the recorded cause, which is no longer the digest-resolution bug fix(ci): resolve nightly packages digest without image-specific skopeo ops #3332 fixed. docs: add CNCF General Technical Review (GTR) questionnaire #3572's GTR questionnaire reports the same flapping independently and adds that e2e-tag.yaml also failed around the v1.6.1 release, which may share a cause. Update 2026-10-02: scheduled nightly was green on 2026-08-17 and 2026-09-03, so the count of failures above is off by those two.
• A17 is partial rather than open. The specific ci: guard against pinned base-image digests disappearing upstream #3305 trigger is fixed: fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3386 re-pinned the deleted digest (2026-07-21) and fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux #3400 moved the base off quay entirely, so packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile:17 is now docker.io/library/almalinux:9@sha256:d2515c76… and no quay.io/centos base remains under packages/. The systemic half, an inventory of volatile digest-pinned bases, a project-controlled mirror, a scheduled preflight that verifies every pinned base still resolves, is untouched, and ci: guard against pinned base-image digests disappearing upstream #3305 stays open on that.
• A11 has gone backwards rather than forwards. test(e2e): add chainsaw-native release upgrade testing lane #3276 is still open, still CHANGES_REQUESTED, and now conflicting (last updated 2026-07-31), so a rebase is part of the work. The jobs it has to sequence against also changed under ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote #3449 (2026-07-29): tags.yaml now runs rc-e2e (line 322) ahead of generate-changelog (368) and update-website-docs (625), and promote-rc.yaml carries its own changelog (499) and website-docs (536).
• A10 was not what closed test(e2e): migrate the E2E app suite from BATS to Kyverno Chainsaw #2979. The app-suite bulk landed under A10, and the two monitoring BATS suites that never ported are now [Tracking] Shell test architecture: vanilla BATS for unit tests, assert library, testable suite shell #3453 item 7.

Ready now, no blockers

• A6, go build ./... gate. The gate exists: codeql.yml runs it in the root module, api/apps/v1alpha1 and the three image modules on every PR since 2026-06-26, and go-module-tests runs go test ./... in the nested modules. CodeQL is not a required check, the required contexts are pre-commit and E2E Tests. Needs a decision: count CodeQL and close ci: add CI gates for codegen drift, Go build, and Go unit tests #2499, add a step to pull-requests.yaml, or make the CodeQL check required.
• A9, install wiring. The design changed: feat(platform, tests): select E2E packages and declare controller dependencies #4022 (open, CHANGES_REQUESTED) adds select-install.sh --disabled <suites>, which emits bundles.disabledPackages for an ordinary variant, so the test-minimal bundle and COZYSTACK_ENABLED_PACKAGES are obsolete. Wiring it into a workflow is follow-up work, nothing under .github/ references select-install. The measurement was read by hand from 62 PR e2e jobs on the container lane (2026-09-28 to 09-30): install took 28 min median, and Run E2E tests was full-shaped in 73% of jobs, scoped in 23% and skipped in 5%, classified by step duration. Install selection can only save on the scoped jobs. The window is three days and heavy on re-runs.
• A12, drop redundant post-HR-Ready waits across hack/e2e-chainsaw/*/chainsaw-test.yaml, find the assert/wait steps that duplicate HR Ready, keep the CR-operator-async waits. The audit in Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653 was redone on the Chainsaw suites on 2026-10-01, the candidate list, the check of the nested -system HR on a dev cluster and the scope of the first PR are in its comments. Coordinate with [Epic] Adopt the kstatus readiness model platform-wide (Flux 2.8 / helm-controller v1.5.0) #2642: feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273 (merged 2026-07-21) landed the typed waitStrategy/healthCheckExprs API, adopted so far only in packages/extra/monitoring and packages/extra/seaweedfs, and feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 is still a draft with changes requested.

Blocked / sequential

• A3, tag-protection rulesets (v*, api/apps/v1alpha1/* creation-restricted to the CI app). Repo-admin only. The repo has no rulesets of its own, the one that applies is an org-level branch ruleset ("Require PR review for default branches", a pull_request rule, no tag target). Ruleset + rollout order documented in docs/release.md via feat(release): add Cut Pre-release Tag workflow (#2677) #3253.
• A4, enable RETENTION_APPLY (retention still defaults to dry-run at .github/workflows/retention.yaml:65). Bundles with A3 in one admin request. The workflow fix is done (fix(ci): resolve nightly packages digest without image-specific skopeo ops #3332), and the red nightly runs are ci(nightly): Mirror images to GHCR fails on ghcr write_package, so the nightly e2e has not run since 2026-09-15 #4423 (mirror job) and A18 (e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642, E2E job). Flipping it would delete nothing today, A19 comes first.
• A2, SBOM: OCIR referrers/attestation probe → --sbom=true → delivery decision. The plumbing already exists (hack/common-envs.mk:61, :71: SBOM=1 → --sbom=true, landed 2026-06-22) and no workflow sets it, so the remaining work is the probe and the delivery decision, not the build flag. Ties into the OpenSSF supply-chain work (ci: harden GitHub Actions workflows for OpenSSF Scorecard (pin, permissions, zizmor gate) #3223) and is cited as a gap in the CNCF GTR questionnaire (docs: add CNCF General Technical Review (GTR) questionnaire #3572).
• A11, upgrade lane (test(e2e): add chainsaw-native release upgrade testing lane #3276). Open, conflicting, CHANGES_REQUESTED on 2026-09-29, and it needs a rework more than a rebase. The review lists: 13 of 19 commits carry trailers the contributing rules don't allow, the lane sits on QEMU and on artifacts the main PR pipeline no longer produces, baseline v1.6.4 has no drbd.enabled in the linstor chart so a baseline run needs QEMU until a release has it, and the lane has its own HR wait next to cozy_wait_all_helmreleases_ready. test(e2e): wire the 1.5→1.6 etcd upgrade job into CI as the authoritative regression guard #3316 was closed as stale on 2026-10-01, it asked for a 1.5 to 1.6 job and v1.6.4 is stable and main is v1.7.0-alpha.3.
• A17 (systemic half), inventory volatile digest-pinned bases, mirror into a project-controlled registry, add a scheduled preflight that verifies every pinned base still resolves. The ci: guard against pinned base-image digests disappearing upstream #3305 trigger itself is fixed (fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3386, fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux #3400). Sibling of chore(maintenance): refresh and automate alpine/k8s hook image pins #2565. A third incident happened: 14 of 16 pinned quay.io/kubev2v digests in packages/system/forklift-operator/templates/forklift-operator.yaml no longer resolve (forklift-operator: 14 of 16 pinned images no longer exist on quay.io #4486). It sits in a vendored template, outside what ci: guard against pinned base-image digests disappearing upstream #3305 looks at (Dockerfiles and .tag files), so the scope has to widen to templates and values.
• A18, retire the QEMU lane and run what only it covered on a temporary three-node cluster (e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642). The design for the temporary cluster is not started. The interim step proposed in the issue, nightly and rc e2e on the container lane the way test(e2e): run the merge-gating lanes on Talos containers #4020 did it for the PR jobs, is not decided and not tried against a published tag. Until one of them lands, nightly.yaml and e2e-tag.yaml stay on QEMU, rc e2e is red and releases go out with skip_e2e_gate: true.
• A19 (new), retention listing. The org package listing in retention.yaml returns gh: Invalid argument. (HTTP 400). It did on 2026-07-08 already, back then the job printed No cozystack/* container packages found and exited 0. ffae003d6 (2026-08-14) made that case fail, so every scheduled run is red since 2026-08-18. I couldn't find the cause of the 400, my token gets 403 on the packages API.

Effort and closeability

Sizes recalculated 2026-08-13 against current state, not carried over. The A6, A9, A11 and A13 rows were updated on 2026-10-02. S = hours, M = a day or two, L = multi-day.

Issue → closeability

• chore(ci): align backport labels with namespaced label convention #2585 closed on 2026-08-27 (A13).
• ci: add CI gates for codegen drift, Go build, and Go unit tests #2499 closes when A6 is decided, the gates exist.
• Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653 closes on A12, one PR.
• RFC: immutable release tags & rc-promotion flow #2677 was closed on 2026-08-14 as implemented: feat(release): immutable tags & rc→stable promotion (#2677) #3017, feat(release): add Cut Pre-release Tag workflow (#2677) #3253, ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote #3449 and feat(release): cutting an rc freezes the line into release-X.Y #3399 landed the flow, and A5 proved it live on v1.6.0 and v1.6.1. A3 and A4 stay tracked here as repo-admin settings, which is why closing the RFC did not wait on them.
• Finish the CI image-build refactor: SBOM emission and pr-* tag retention #2937 closes on A2 plus the physical runner retirement and the rendezvous rehome (the A1 residual). A17 is adjacent rather than a closure gate.
• [Epic] Diff-driven & scenario-based E2E testing for Cozystack #2847 closes on A9 (Track A) + A11 (Track B), and A8, A10 and A16 are done. A9 is feat(platform, tests): select E2E packages and declare controller dependencies #4022 plus the workflow wiring now.
• test(e2e): lightweight E2E cluster (kind / Talos-in-Docker) for diff-scoped PR runs #3238 was closed on 2026-10-01 as done via test(e2e): run the merge-gating lanes on Talos containers #4020 (A14 and A15).
• e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642 closes on A18.
• ci: guard against pinned base-image digests disappearing upstream #3305 (A17) closes on the systemic inventory + preflight, not on the almalinux swap that unblocked the build.
• test(e2e): migrate the E2E app suite from BATS to Kyverno Chainsaw #2979 (Chainsaw migration) was closed on 2026-08-14. The two unported hack/e2e-apps/monitoring-oidc-*.bats suites moved to [Tracking] Shell test architecture: vanilla BATS for unit tests, assert library, testable suite shell #3453 item 7 as a coverage regression rather than a migration remnant, and neither is covered by an action ID here. They were deleted in test(e2e): remove unused monitoring suites and audit BATS runners #4019 instead of being ported.

Recommended sequence

1. Fix A19, then fire off the admin request for A3 + the RETENTION_APPLY flip in A4 (longest external lead time).
2. Decide A6 and close ci: add CI gates for codegen drift, Go build, and Go unit tests #2499.
3. One Chainsaw read-through for A12.
4. A9: get feat(platform, tests): select E2E packages and declare controller dependencies #4022 green and reviewed, then wire select-install.sh --disabled into the PR and fork lanes.
5. A2 SBOM (probe → enable → delivery decision).
6. A11 once A18 says which substrate the upgrade lane runs on, then A17's preflight.
7. The nightly lane's red runs are ci(nightly): Mirror images to GHCR fails on ghcr write_package, so the nightly e2e has not run since 2026-09-15 #4423 (mirror job, being worked separately) and A18 (e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642, E2E job).