bug(tests): the HelmRelease waits and helpers the HR-Ready raise could not reach #4470

The change that closes #3771 raises every Chainsaw HR-Ready assert and every suite-driven wait_hr_ready above the install timeout of the release it waits on, except kafka's, whose 7h cannot be cleared inside the E2E job. Some waits of the same class are out of its scope, because changing them touches files that change does not own, and nothing yet holds the raised numbers in place. Line numbers are on main at 564f540.

Tenant addon waits in hack/e2e-chainsaw/_lib/run-kubernetes.sh

The kubernetes suites wait on the addon releases the kubernetes chart renders into the tenant, and each of those templates sets spec.timeout: 10m with no spec.install.timeout:

• :5679 waits 10m on kubernetes-<name>-csi (packages/apps/kubernetes/templates/helmreleases/csi.yaml:12), level with its floor.
• :5805 waits 5m on each of cilium, coredns, csi and vsnap-crd (cilium.yaml:54, coredns.yaml:29, csi.yaml:12, volumesnapshot-crd.yaml:23).
• :5807 waits 5m on ingress-nginx (ingress-nginx.yaml:53).
• :5815 waits 10m on the optional ouroboros (ouroboros.yaml:135), level with its floor.

These are not a straight copy of the app-suite case, and I have not worked out how far under they really are. The addon installs start when the parent release renders them, well before these waits run, so the wait's clock starts late rather than early. Whether a 5m wait expires before a failed addon install records its reason depends on how much of that install has already run when the wait begins, which the script does not measure. Deciding the numbers needs that measured, or a wait sized from the addon's creation rather than from the step.

Any edit to _lib/ escalates the whole E2E run, so this is worth batching with other work on that file. #3773 covers the missing backstops on some of the same lines. That is a different defect, but the same edit could close both.

Two wait_hr_ready helpers print no reason on timeout

examples/backups/redis/00-helpers.sh:102 and examples/backups/kafka-metadata/00-helpers.sh:187 print only Timeout waiting for HelmRelease/<name> to become Ready: no condition, no reason, no release history. The copy shared by clickhouse, mariadb, mongodb, postgres and rabbitmq prints every condition with its reason, the release history, and separate lines for a release that never appeared and for a kubectl that could not answer. Neither of these two is a copy of that body, so they got none of it. hack/mariadb-first-boot-waits.bats lists both as named exceptions to its byte-identity check. Fixing them does not depend on any budget: a timeout becomes readable whatever the number is. The kafka-metadata one matters most, because its 300s waits sit under a 7h floor by construction, so its timeout message is the only diagnosis a failed install there will ever get.

Opt-in etcd round-trip

examples/backups/etcd/02-create-bucket.sh:29 and 03-create-etcd-src.sh:29 use a bare kubectl wait hr ... --timeout=300s on generated releases with the 600s floor. hack/e2e-chainsaw/etcd/chainsaw-test.yaml drives that example only when ETCD_E2E_S3_ROUNDTRIP=1, so CI never runs it today. A bare kubectl wait also prints nothing about the release on timeout. The nats and foundationdb examples carry the same 300s kubectl wait hr calls, but no suite drives them.

Nothing holds the raised numbers

hack/mariadb-first-boot-waits.bats derives the floor and holds the waits against it for the suites in its MBW_SUITES table and for the mariadb Chainsaw asserts. Every other raised Chainsaw assert, and the redis and kafka walkthrough bucket waits, can drop back under its floor without anything turning red. Extending that derivation to them would keep the relation true when either side moves, which is what #3771 originally suggested.

A parent release can give up before its nested child records why

harbor-test is generated with a 600s install timeout, and its install waits through kstatus on the nested harbor-test-system, whose own install timeout is 15m (packages/apps/harbor/templates/harbor.yaml:91). When the child hangs, the parent fails at about 10m with the child still InProgress, the parent's assert fails its step, and the child's assert never runs; the child records its own reason only at 15m. external-dns, openbao and qdrant have the same shape, a 600s parent over a 10m child whose clock starts later. The test budgets are right for the parent's own reason; fixing this means aligning the chart timeouts so a child finishes, or fails, inside its parent's budget.

How to find them again

grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/ examples/ for the plain waits, and grep -rn 'wait_hr_ready "' examples/backups for the helper calls. Read the helper's default for any call that passes no budget.