bug(tests): most app suites give up on a HelmRelease before it can record why its install failed #3771

Fourteen Chainsaw suites assert HelmRelease readiness at or below the install timeout of the release they are waiting on, so the assert always expires first and the reason the install failed is never reached. pkg/cmd/server/start.go:91 sets HelmReleaseInstallTimeout: "10m" for every release cozystack-api generates, no deployment passes --helmrelease-install-timeout, and only four ApplicationDefinitions override it (kubernetes at 20m, tenant, kubernetes-nodes and monitoring at 15m), none of which these suites use.

The budget shown is the tightest HelmRelease-ready assert in each suite; several carry more than one. mariadb is excluded from the list because its budgets are the subject of a separate change.

The list is worth re-deriving rather than trusting, and the derivation is the point: walk every hack/e2e-chainsaw/*/chainsaw-test.yaml, find each kind: HelmRelease inside an assert:, take the nearest enclosing timeout:, and compare it against that release's floor. The neighbouring list, of the kubectl wait calls on HelmReleases elsewhere in hack/, resists being written down at all: a grep for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the --all ones. The table below survives both traps because the asserts it walks are structurally uniform, but a reader adding a suite should re-run the walk rather than extend the rows.

Two suites look like the worst cases and are not cases at all. kubernetes-oidc-customconfig and kubernetes-oidc-system assert at 1m, but both wait on releases generated from the kubernetes ApplicationDefinition, which sets release.cozystack.io/helm-install-disable-wait: "true" (packages/system/kubernetes-rd/cozyrds/kubernetes.yaml:28). With the wait disabled the release goes Ready as soon as helm has applied, so 1m is a budget for an apply and is fine. kubernetes-nodes is the only other definition that disables it. Anyone sorting this list by budget will reach for those two first, which is why they are named here rather than left out.

harbor needs its own note, in the opposite direction: its parent release is not generated by cozystack-api but rendered by the app chart, which sets spec.timeout: 15m (packages/apps/harbor/templates/harbor.yaml:91). Its 10m assert is therefore five minutes under its real floor, not level with it as a comparison against the 600s default would suggest.

What this costs

When an install genuinely fails, Chainsaw reports its own timeout against a release that is still mid-install. The release records why the install failed when its own timeout fires, which is after the assert has already given up. The suite says the HelmRelease did not become Ready in six minutes, which is true and carries no information about why.

This is not a flake. These suites do not go red more often than they should; they go red just as often and say less when they do. Nothing here changes how many runs fail, only what a failed run can tell you.

I checked packages/apps/{clickhouse,redis,mongodb,postgres,foundationdb}/templates/ for a startupProbe and found none, so for those five there is no first-boot budget hiding under the wait either. I have not extended that check to the other nine, so whether any of them also needs headroom for its own bootstrap is open.

Why the ordering matters at all

helm-controller leaves spec.waitStrategy at its default of poller, so the install waits through kstatus, which falls through to a Ready-condition fallback for kinds it has no rules for and maps Ready=False onto InProgress. Any of these releases that renders a custom resource publishing a Ready condition therefore keeps the install running until that resource is healthy or the install's own budget is spent. The assert in front of it is sampling a release that is still working, and the only thing it can report is that the release had not finished.

Suggested shape

Whatever budget these asserts end up with should sit above the release's install timeout rather than at or below it, so a failed install has recorded its reason before the assert gives up.

One piece of prose has to move with the numbers. hack/e2e-chainsaw/mariadb/chainsaw-test.yaml:25 justifies its own budget as "11m rather than the usual 5-6m". That is descriptive of the table above and stops being true the moment this list is worked through, at which point it reads as an endorsement of the band it was contrasting against. Whoever raises these budgets owns that sentence.

The general form has two floors and a wait has to beat the larger. The first is the app's own first boot, which for a chart that sets startupProbe is initialDelaySeconds + (failureThreshold - 1) * periodSeconds and for a chart that does not is whatever its operator needs. The second is the install timeout, which is the binding floor whenever the first is smaller, and which is not one number: it is 600s by default, 900s or 1200s where an ApplicationDefinition overrides it, the chart's own spec.timeout for a release the chart renders, and absent entirely where the wait is disabled. Deriving it from the tree rather than pinning a literal is what keeps the relation true when either side moves; hack/mariadb-first-boot-waits.bats does that for one suite and is the shape to copy.