fix(tests): let the mariadb waits cover a first boot - #3769

What this PR does

Closes #3741

Several waits for a database HelmRelease to become Ready gave up before the release could: the mariadb, mongodb, rabbitmq and clickhouse backup walkthroughs waited 300 seconds, and both mariadb Chainsaw suites gave their HelmRelease-ready assert 6 minutes. A generated release carries a 10-minute install timeout (pkg/cmd/server/start.go:91), so a failed install was reported as a bare timeout, because the release had not yet recorded why it failed, and for mariadb a first boot the startup probe was still willing to wait for failed the harness instead.

A wait on such a release has two floors and has to beat the larger. The first is the app's own first boot: cozystack-api leaves spec.waitStrategy unset and helm-controller defaults it to poller, so the install waits through kstatus, and kstatus maps the operator CR's Ready=False onto InProgress. For mariadb that boot may take the chart's 310-second startup budget (startupProbe.failureThreshold: 30 in packages/apps/mariadb/templates/mariadb.yaml with the operator's probe defaults of 20 and 10); mongodb, rabbitmq and clickhouse set no startup probe. The second is the install timeout. At or below it the waiter is the one that expires, because its clock starts when the step runs and the install's only once the Application has been reconciled and the chart artifact fetched. The install timeout binds in every case here, so the walkthrough waits go to 660 seconds and the mariadb asserts to 11 minutes.

Raising a wait inside a walkthrough spends the Chainsaw op that runs the script, and an op that runs out SIGKILLs the process group and loses the script's error. Each op therefore grows by exactly what the waits it reaches gained: mariadb's round-trip from 30 to 42 minutes, rabbitmq's from 30 to 42, mongodb's from 30 to 36 (its round-trip runs with SKIP_RESTORE, so only the source wait counts), and clickhouse's from 60 to 72, which still sits above the sum of every inner ceiling. That bounds a failure at a single wait, not a run that is slow at every stage and then fails late; bounding that needs a phase budget, not a larger literal. postgres keeps its 360-second waits and every walkthrough keeps its 300-second bucket wait: whether a suite pays the extra minutes on every red run is a per-suite call.

The mariadb endpoint asserts keep their job. A freshly applied CR carries no conditions and kstatus reads it as current, so the install wait can end before any pod is up; an endpoint address closes that window. That is also why the wait_for_field on the CR stays in the walkthrough.

wait_hr_ready now dumps every condition and the release history on timeout and on Stalled=True, instead of the single Ready message a scheduled retry can overwrite before anyone reads it. It also stops printing one (none recorded) line for different answers: a release that never appeared, a release deleted during the wait, a lookup kubectl could not answer, a failed history read and an empty history each say so, with kubectl's own stderr kept, and a release seen earlier still gets its dump when only the last lookup failed. The change goes to every byte-identical copy of the helper, in examples/backups/{clickhouse,mariadb,mongodb,postgres,rabbitmq}/00-helpers.sh, and they stay identical.

hack/mariadb-first-boot-waits.bats pins the mariadb budgets against the tree. It derives both floors from their sources instead of pinning literals, so raising failureThreshold past the install timeout, or changing the install-timeout default, reds it; each source has a test that refuses to proceed when it cannot be read, including the ApplicationDefinition, whose two override annotations would each invalidate the derivation. It holds the round-trip op at a fixed remainder over the ceilings up to and including the source wait, bucket stage and S3 preflight included, and at a fixed slack once both mariadb waits are paid, so raising a wait without the op reds. The extractors are also tested against fixtures, because the tree cannot tell a correct extractor from a broken one. Beyond mariadb, the file runs every exit branch of wait_hr_ready in each copy against a stubbed kubectl, and pins the copies byte-identical. The population is every definition in the tree found by grep, minus two named exceptions held to still differ, so a new copy joins without editing the test and an exception whose copy converges goes red:

• examples/backups/redis/00-helpers.sh is written in the compact one-line style of the rest of that file, so converging it rewrites its shape rather than applying the shared body.
• examples/backups/kafka-metadata/00-helpers.sh is a helper of its own with the same diagnostic gap on timeout (it prints no reason at all), and closing that is a change to kafka-metadata.

docs/agents/e2e-testing.md replaces the reviewer checklist's 5-6m standard and its list of exceptions, which the tree had outgrown, with the floor a wait has to beat and the rule that a longer wait carries its reason in-line. The derivation gets a section of its own, opening with the rule in one line. For a release an app chart renders itself the floor is spec.install.timeout, then spec.timeout, then helm-controller's 5m default, and the nested <name>-system releases never take cozystack-api's 10m. The harbor comments name the floor each of their asserts sits against, and the opensearch comment stops citing the removed standard.

Adding hack/mariadb-first-boot-waits.bats does not escalate the E2E selection: hack/[^/]+\.bats$ sits in inert_config_pattern in hack/select-e2e.sh, and full_suite_pattern escalates only hack/e2e-*.bats. On this changeset hack/select-e2e.sh selects clickhouse harbor mariadb mongodb opensearch postgres rabbitmq.

What this costs is paid only on a failing run; the raised numbers are ceilings. Summing the timeouts stated on each suite's own steps, mariadb goes from 92m to 114m, clickhouse from 116m to 128m, rabbitmq from 67m to 79m and mongodb from 95m to 101m, before the untimed steps that fall back to the defaults in .chainsaw.yaml. clickhouse is the most expensive suite by that sum. All of them compete for the 215m timeout-minutes of the whole e2e job, which also covers image download, sandbox preparation and the Cozystack install, and a job cancelled at the cap loses both its log and cozyreport.tgz.

This does not fix the failure that led here. In the run that prompted the investigation the mariadb replica never opened port 3306 at all, spending its whole 310-second budget inside mariadb-install-db before being killed, and no wait budget would have saved it. That defect is #3744. What this fixes is a second, independent defect found while reading the same code.

One thing here is argued rather than measured: the mechanism is checked against helm-controller and kstatus, and every number is derived from the tree, but no run ID pins that it is what any particular red run hit. Both floors justify 660s whatever the cause was, so being wrong about the cause leaves slack rather than breakage.

Not in this PR

• This PR retires the mariadb, mongodb, rabbitmq and clickhouse walkthrough waits and the two mariadb asserts. The rest is tracked in test(e2e): HR-Ready waits sit under the install timeout of the release they wait on #4404: the other Chainsaw HR-Ready asserts under their floor, the harbor asserts, the 300-second bucket waits, the postgres waits, and the redis and kafka-metadata diagnostic gap.
• The clickhouse round-trip comment says every wait fails fast, wait_hr_ready on Stalled=True, "so a genuinely broken backup reports in seconds". A release whose install is still running is neither Ready nor Stalled, so that holds for a stalled release and not for an install that has not finished. The sentence predates this change.
• hack/mariadb-first-boot-waits.bats is large for what it guards (two floors, two ratchets on the op, and the helper identity pin), with many fixture tests on its own extractors. Trimming it to that core is a separate decision.
• Two comments in that file state facts about the mariadb suite that are true today and that nothing checks: three script ops carry a minute timeout, and the only one ahead of the round-trip is the 8m pre-clean.
• The mariadb op comment says a bound on the whole flow "needs a phase budget (docs/agents/e2e-testing.md)"; the doc describes it as bounding the collection phase as a phase (COZY_DIAG_PHASE_BUDGET), so the pointer would resolve faster if it named that bullet.
• Cosmetic: the behavioural test's loop body is not indented under its for, one line of the bats header runs past the width of the rest, and the checklist's sentence under the example assert ("what this block demonstrates least") reads awkwardly.

Screenshots

Not a UI change.

Downstream repositories

Walked the trigger map in docs/agents/contributing.md against the diff, file by file, for all eighteen files. The five examples/backups/*/00-helpers.sh files change the diagnostics of wait_hr_ready, and the mariadb, mongodb, rabbitmq and clickhouse walkthrough scripts change wait budgets; the manifests the walkthroughs apply and the flows they document are unchanged, so no website page restates anything that moved. Four Chainsaw suites (mariadb, clickhouse, mongodb, rabbitmq) change timeouts and comments, and the harbor and opensearch suites change comments only. docs/agents/e2e-testing.md changes the reviewer checklist entry and adds the sizing section. hack/mariadb-first-boot-waits.bats is a new file picked up by the existing BATS_UNIT_FILES wildcard. hack/package.mk, hack/common-envs.mk and hack/update-crd.sh are untouched, and no values.schema.json, values.yaml default, version enum, ApplicationDefinition, CRD, namespace or release asset is touched.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

fix(tests): the MariaDB, MongoDB, RabbitMQ and ClickHouse backup examples and the MariaDB Chainsaw suites now wait for a HelmRelease past the release's own install timeout, so a failed install reports its own reason instead of a bare timeout, and the examples' HelmRelease wait now tells a release that never appeared, one deleted mid-wait and a kubectl that could not answer apart.