e2e: the parent-HR guard comment refutes itself, and eight waits in the file have no backstop #3773

Two pre-existing problems in hack/e2e-chainsaw/_lib/run-kubernetes.sh, both about the parent-HelmRelease remediation guard and the wait above it. Neither is introduced by any open PR; both are on main today. The second one turned out to be wider than the single line it was first spotted on, so the scope below is the file, not the line.

The guard's own comment states what the rest of the same comment refutes

Line 2274 introduces the guard with this:

# A non-zero installFailures/upgradeFailures indicates the helm-wait budget expired while
# admin-kubeconfig was still being provisioned, which would trigger uninstall remediation
# and churn the Cluster CR.

Line 2282, eight lines further down in the same comment block, says the opposite:

# ... unlike the installFailures/upgradeFailures
# counters (which ClearFailures zeroes on every successful reconcile).

The second statement is the correct one, and it is the reason the guard reads .status.history instead. A reader who stops at 2274 walks away with exactly the belief the block below exists to remove, and docs/agents/e2e-testing.md spells out the same thing under "Assert the parent HelmRelease did not remediate": checking those counters after the HR is Ready is vacuous and passes against a reverted fix.

The fix is to drop the counter sentence from the opening lines, or rewrite it to say the counters are what the guard deliberately does not use.

Waits without an event-driven backstop, eight candidates rather than one

AGENTS.md requires an event-driven backstop before every kubectl wait. Line 2271 has none:

kubectl wait hr -n tenant-test "kubernetes-${test_name}" --timeout=5m --for=condition=ready

That line is where this was noticed, but it is not alone. The file holds 21 waits; 9 go through the kubectl_wait_retry helper and 8 are bare:

1603  wait namespace tenant-test --for=jsonpath={.status.phase}=Active
1934  wait hr ...-csi --timeout=10m --for=condition=ready
2003  wait pvc nfs-test-pvc --for=jsonpath={.status.phase}=Bound
2049  wait machinedeployment ...-md0 --for=jsonpath=...readyReplicas=2
2052  wait hr ...-${component} --timeout=5m --for=condition=ready
2054  wait hr ...-ingress-nginx --timeout=5m --for=condition=ready
2062  wait hr ...-ouroboros
2271  wait hr ... --timeout=5m --for=condition=ready

Counted by call form (grep -nE "^\s*kubectl (--kubeconfig [^ ]+ )?wait " over the file as it stands on main), not by checking each site for a backstop earlier in the function. So eight is an upper bound on candidates, not eight confirmed violations: some of them may already have one above. Whoever picks this up should check per site.

The number matters because an issue written about one line gets closed by fixing one line, and the next timeout in this file reports just as little as this one does.

The other nine are not a safe remainder either, for a separate reason that already has its own issue: kubectl_wait_retry captures output as _out=$(kubectl wait "$@" 2>&1) and reads _rc=$? on the next line, so under errexit the shell dies on the assignment and the retry logic below it is unreachable. The library does not set errexit itself; the caller does, in the chainsaw script step (kubernetes-latest/chainsaw-test.yaml:56 is set -eu immediately above the . run-kubernetes.sh that step sources). Stated that way so checking the library in isolation does not read as "cannot reproduce". That part belongs to #3684, not here.

Both problems above live in the same eighteen lines, so one change covers them; the wider backstop sweep can follow in the same PR or a separate one.