bug(tests): five app suites give up on a HelmRelease before it can record why its install failed #3753

clickhouse, redis, mongodb, postgres and foundationdb each assert HelmRelease readiness below the release's own install timeout: hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml:29, hack/e2e-chainsaw/redis/chainsaw-test.yaml:25, hack/e2e-chainsaw/mongodb/chainsaw-test.yaml:25 and hack/e2e-chainsaw/postgres/chainsaw-test.yaml:32 at 6m, hack/e2e-chainsaw/foundationdb/chainsaw-test.yaml:34 at 5m. The release they wait on carries a 10-minute install timeout: pkg/cmd/server/start.go:91 sets HelmReleaseInstallTimeout: "10m", no deployment passes --helmrelease-install-timeout to cozystack-api, and none of the five ApplicationDefinitions sets the release.cozystack.io/helm-install-timeout override.

So the assert always expires first, by four or five minutes. When an install genuinely fails, Chainsaw reports its own timeout against a release still mid-install, and the reason the install failed, which the release records when its own timeout fires, is never reached. The suite says the HelmRelease did not become Ready in the budget it was given, which is true and carries no information about why.

What this is not

This is not a first-boot flake. That needs a chart whose startupProbe budget can outlast the wait in front of it, and none of these five sets startupProbe at all. I checked packages/apps/{clickhouse,redis,mongodb,postgres,foundationdb}/templates/. They are exposed only to the diagnostics loss described above, so raising their budgets to cover a first boot would be fixing a problem they do not have.

The failure mode is narrower and duller than a flake, and worth separating for that reason: these suites do not go red more often than they should. They go red just as often and say less when they do.

Why the ordering matters at all

helm-controller leaves spec.waitStrategy at its default of poller, so the install waits through kstatus, which falls through to a Ready-condition fallback for kinds it has no rules for and maps Ready=False onto InProgress. Any of these releases that renders a custom resource publishing a Ready condition therefore keeps the install running until that resource is healthy or the install's own 10 minutes are up. The assert in front of it is sampling a release that is still working, and the only thing it can report when it expires is that the release had not finished.

Suggested shape

Whatever budget these asserts end up with, it should sit above the release's install timeout rather than below it, so that a failed install has recorded its reason before the assert gives up.

The general form has two floors, and a wait has to beat the larger of them. The first is the app's own first boot, which for a chart that sets startupProbe is initialDelaySeconds + (failureThreshold - 1) * periodSeconds and for a chart that does not is whatever its operator needs; none of these five sets one. The second is the install timeout above, which every generated release carries and which is the binding floor whenever the first is smaller. Deriving both from the tree rather than pinning a literal keeps the relation true when either side moves.

Whether these five also need more headroom for their own bootstraps is a separate question this issue does not answer. That would need a measurement per chart, and I have not made one.