test(e2e): let HR-Ready waits outlast the install timeout of the release they wait on - #4471

What this PR does

Stacked on #4468, which holds the raised backup waits to a start margin; review that one first. This PR's own diff is the six commits on top of it.

Most HR-Ready waits in the E2E suites gave up before the release they were waiting on could fail its install. cozystack-api gives every release it generates a 600s install timeout, and the Chainsaw asserts in front of those releases sat at 5 or 6 minutes. When an install genuinely failed, the suite reported a bare assert timeout against a release that was still installing, and the reason the install failed, which the release records when its own timeout fires, was never reached. Red runs failed just as often as they should; they only said less.

I raised every HR-Ready wait above the install timeout of its own release, by the 60s start margin #4468 introduces:

• Chainsaw asserts on releases cozystack-api generates go to 11m, across bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance.
• Asserts on nested <name>-system releases take their floor from the template that renders them: 11m for external-dns, openbao and qdrant (10m spec.timeout), 16m for harbor-test-system (15m). harbor-test sat exactly on its 600s floor, which expires first because the assert's clock starts before the install's; it goes to 11m too.
• The kuberture Package release takes its 10m from cozystack-operator and goes to 11m.
• In the backup walkthroughs, the bucket waits go from 300s to 660s in clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and the two postgres waits go from 360s to 660s. The kafka bucket wait passed no budget and silently took the helper's 300s default, which is why earlier scans for literal budgets missed it. Each Chainsaw op that runs a walkthrough grows by exactly what its waits gained (clickhouse 78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m, redis 36m), so a failure at any later wait keeps the headroom it had and is still reported by the script instead of being cut off by the op.

Two waits stay as they are, on purpose:

• kafka: kafka-rd sets a 7h install timeout, which no budget inside the 215-minute E2E job can clear. The kafka assert and the Kafka waits in the kafka and kafka-metadata walkthroughs keep their numbers, and the assert now says why in-line.
• vm-instance-test: vm-instance-rd disables the install wait, so its release goes Ready as soon as helm has applied and there is no floor to clear. Earlier lists of this problem included it by mistake.

The cost falls on red runs only: a suite whose release never goes Ready now spends up to six more minutes before it reports, and in exchange the report carries the release's own reason. Green runs are unchanged. Suites run one after another in a single job capped at 215 minutes, so a systemic failure that reddens many suites now reaches that cap sooner, and a job cancelled at the cap loses its log and cozyreport.tgz. I took that trade because a failed install that says why is worth more than a red run that ends a few minutes earlier.

docs/agents/e2e-testing.md stopped being true in several places once the waits were raised: it described most of the tree as under the floor, mariadb as the only suite raised, the walkthrough bucket waits as under it, and harbor's asserts as sitting on and under theirs. I rewrote those sentences, and the one that pointed at "every wait above the band" of 5-6m, kept the start-margin wording from #4468, and added that a walkthrough wait passing no budget takes the helper's default. The comments in hack/mariadb-first-boot-waits.bats that quoted the old op and bucket figures are restated; its existing checks and constants are unchanged, because every op grew by exactly its bucket delta and each remainder still sits at exactly its constant. I also added postgres to the table that file holds the walkthroughs to, so its raised waits and its op cannot drift back apart unnoticed, and a test that holds each table suite's bucket wait to the bucket release's floor, read from the api-server default with bucket-rd checked for an override; the op check alone could not catch a bucket wait dropping back, since a lower wait only widens the slack. redis stays out: its walkthrough waits on no redis HelmRelease, only on the bucket and on the sentinel master, so the table's floor check has nothing to read there.

How I checked it, since E2E cannot run locally:

• I re-ran the enumeration from the sizing section of docs/agents/e2e-testing.md over every hack/e2e-chainsaw/*/chainsaw-test.yaml and every wait_hr_ready call under examples/backups/, literal or default, and held each against its own release's floor. Every Ready=True assert and every wait_hr_ready call a suite drives now clears its floor, except the kafka ones named above. The opt-in etcd round-trip uses bare kubectl wait hr calls at 300s instead of that helper; they are listed in bug(tests): the HelmRelease waits and helpers the HR-Ready raise could not reach #4470. The Ready=False assert on the rejected redis upgrade is not a readiness wait and is left out.
• make bats-unit-mariadb-first-boot-waits passes, 31 of 31. For the new postgres row, putting the op back to 50m fails only the postgres slack test, and a postgres wait at 659s fails only the postgres floor test. Setting any one table suite's bucket wait to 659s fails only the bucket test, naming only that suite, and an install-timeout override on bucket-rd fails it too. I ran the mutations under plain bats so a fail-fast runner could not hide a second failure.
• hack/select-e2e.sh over this diff selects bucket, clickhouse, etcd, external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance.

Out of scope, and tracked in #4470: the tenant addon waits in hack/e2e-chainsaw/_lib/run-kubernetes.sh (an edit there escalates the whole E2E run), the redis and kafka-metadata wait_hr_ready helpers that print no reason on timeout, and the opt-in etcd round-trip's bare kubectl wait calls.

Known and left for later, none of them changed here:

• Only the mariadb Chainsaw asserts and the walkthroughs in the guard's table are held by a test. The other raised asserts, and the redis and kafka bucket waits, rest on literals with in-line reasons; bug(tests): the HelmRelease waits and helpers the HR-Ready raise could not reach #4470 covers extending the guard.
• The Kafka waits in examples/backups/kafka/04-create-kafka.sh, 07-restore-to-copy.sh and kafka-metadata/run-all.sh stay at 300s against the 7h floor without an in-line reason at the call site. The doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw assert carries it.

The postgres waits are the ones #3748 is about. Their budget now clears the install-timeout floor and a test holds it there; what a CNPG first boot costs on this platform is still unmeasured, so that issue stays open.

Closes #3771

Closes #3753

Screenshots

Not a UI change.

Downstream repositories

I walked the trigger map in docs/agents/contributing.md against the diff. It touches E2E suites, backup walkthrough scripts, a unit-lane bats file and an agent doc, none of which any downstream repository restates.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout.

Summary by CodeRabbit

• Tests

  • Extended readiness and backup round-trip test timeouts across multiple services to accommodate slower startup and provisioning.
  • Added checks to verify that bucket readiness waits meet configured installation-timeout requirements.

• Documentation

  • Updated testing guidance to clarify how readiness waits relate to release-specific timeouts and overall test budgets.