e2e: an install-phase failure names the wrong waiter and captures nothing that explains the blocked release #3680

The install phase of the e2e suite fails in a shape that is hard to act on: one HelmRelease stalls, every other release reports itself blocked on it, and the job exits without any evidence about why the first one stalled.

Three runs, all failing at minute 19 of the e2e job: #3627, #3607, #3599.

The exit code does not come from the wait most of the output is about. In hack/e2e-install-cozystack.bats, kubectl wait hr --all -A --timeout=15m runs in the foreground and prints 75 to 88 lines of timed out waiting for the condition on helmreleases/<name>, which is what a reader sees first. It is suffixed with || true, so it cannot fail the test at all. Meanwhile hack/e2e-post-install-prep.sh runs in the background waiting for the linstor HelmRelease, times out, prints post-install-prep failed, and it is that path that produces exit 1. The loud waiter is muted by construction; the quiet one owns the exit.

The long list of unready releases is a cascade, not a set of independent failures. Every line carries dependency '<name>' is not ready, and the chain reads end to end: kubeovn blocks piraeus-operator-crds, which blocks piraeus-operator, which blocks linstor, which blocks cozystack-scheduler, while kubevirt, kamaji, metallb and the cluster-api releases block on kubeovn directly.

The blocking release is not the same one every time, so this is at least two failures sharing a phase. In two runs it is cozy-kubeovn/kubeovn, with Last Attempted Release Action: install and Last Attempted Release Action Duration: 10m1.36s, followed by InstallFailed on its own 10-minute Helm timeout. In the third it is victoria-metrics-operator reporting Helm upgrade failed with its operator container restarted six times, while kubeovn there is Ready.

A healthy run converges the whole HelmRelease set in about ten and a half minutes of the Install Cozystack test: #3353 and #3208 finish it at 11:02 and 10:36, and neither reports an InstallFailed for any release. Per-release timings are not available from that output: kubectl wait hr --all walks the objects in order and prints condition met when the walk reaches each one, so a release that was ready long before its line was printed is indistinguishable from one that just became ready. In a passing run the walk sits on one release for six minutes and then prints fifteen lines in two seconds, which is the shape that gives it away. What the failing runs add is a release that reports InstallFailed after exhausting a 10-minute install, which the passing runs never do.

The first four minutes are not where the difference lives. Measured by markers inside the logs rather than by step duration, a failing run reaches Run test: Install Cozystack at 4:10 against 3:40 and 3:48 in the passing ones, and starts its kubectl wait 27 seconds into the test against 26 and 27. Sandbox creation and image pulls ran at the same speed, so a uniformly slower runner does not explain this; the divergence starts below that.

What the phase does not capture is the reason this cannot be settled today. During install nothing collects kubectl top, node allocatable, or kubectl describe node, and the failing logs contain no OOMKilled, no Insufficient cpu and no FailedScheduling. That is not evidence that resources are fine. It is evidence that resource state is not among the things collected. A resource explanation can be neither confirmed nor ruled out until the install phase captures node and pod state the way the suite phase does.

A fourth run, #3608, adds two things. It failed at minute 35 rather than 19, so the phase does not have one characteristic duration. And its blocking release, linstor-scheduler, reports Helm upgrade failed even though this is a first install: the event stream carries exactly one InstallFailed, then UpgradeFailed seventeen times over the following nine minutes. The install failed once, the release stayed in storage, and every later attempt is an upgrade. The Helm output agrees, showing Patched resource rather than Created resource via patch on every cycle. So retry shape and root cause are separate questions here, and the first InstallFailed is the one worth explaining.

Except it cannot be explained from what is collected. The error is empty at both ends: Helm reports warning: upgrade failed: {"error":{},"name":"linstor-scheduler"}, an error object with no fields, and the Kubernetes events carry the same emptiness seventeen times over. Whatever made the first install fail is recorded nowhere in the artifacts, which is the same gap as the missing resource state above, arriving from the other direction: the phase collects the fact of failure and not its reason.

One observation is recorded here without a claim attached. All three failing runs show kube-controller-manager and kube-scheduler on the first node restarted twice, each exiting with unable to load configmap based request-header-client-ca-file against https://127.0.0.1:7445, roughly ten seconds after start. It cannot be compared against a passing run: the previous-instance log capture only runs on failure, so its absence from a green log means the collector never ran, not that the restarts never happened. Deciding whether this is ordinary bootstrap noise or part of the failure needs either a capture on success or a live probe.