e2e: a failed kubectl wait in the Chainsaw library reports nothing, and the retry wrapped around it never runs #3684

kubectl_wait_retry in hack/e2e-chainsaw/_lib/run-kubernetes.sh exists to absorb transient apiserver errors (an etcd leader change, a request timeout, a watch channel closed without a reason) and to print what kubectl wait said when the failure is real. Under the shell options the Chainsaw steps set, it does neither in four of its five call sites.

The helper opens with:

_out=$(kubectl wait "$@" 2>&1)
_rc=$?

The step content runs under set -eu. An assignment whose value comes from a command substitution is itself a simple command, and its exit status is the status of the substituted command, so a failing kubectl wait ends the shell on that line: before _rc is read, before the transient-error branch is consulted, before the printf that would emit the captured text. The output is discarded with the process.

A throwaway reproducer, with the two cases in separate scripts so the harness cannot supply the effect being tested:

# t-bare.sh
set -eu
helper() { local _out _rc; _out=$(false 2>&1); _rc=$?; echo "RETRY-LOGIC-REACHED rc=$_rc"; return "$_rc"; }
echo before
helper
echo after

before
exit=1          # RETRY-LOGIC-REACHED is never printed

Replace the bare helper with if ! helper; then ...; fi and the same script prints RETRY-LOGIC-REACHED rc=1, reports the failure to the caller and continues. That difference is the whole mechanism, and it is also why this is not a guess: the library's fifth call site, at line 408, does sit in an if ! condition, and its retry works. The four bare call sites at lines 1085, 1090, 1093 and 1103 do not.

Two things are lost. The retries never happen, so the transient signatures the helper enumerates are handled nowhere. And nothing says what timed out: Chainsaw receives a non-zero script with no output and prints the only thing it has, ($error == null): Invalid value: false: Expected value: true.

The second loss is the expensive one, because it is not confined to one suite. Every wait failure in this library looks identical in the log regardless of which resource, which condition or which budget expired. A reader cannot tell a MachineDeployment that never scaled from a control-plane Deployment that never became available, and a triager cannot group failures that have no distinguishing text. Some failure signatures that look unfamiliar are not new; the log simply cannot carry what they were.

A worked example, run 31288355964: the kubernetes-previous suite fails after 5m51s with only that generic line. The step's stdout ends at the four deployment ... condition met lines, and in the passing kubernetes-latest suite of the same run the next line is machinedeployment ...-md0 condition met. The next library call is kubectl_wait_retry machinedeployment ...-md0 --timeout=5m --for=jsonpath='{.status.replicas}'=2, and 51 seconds of preamble plus that 5-minute budget is the observed 5m51s. None of that is in the log; it has to be reconstructed from a passing run and the source.

The Error from server (NotFound) lines on stderr in that window are unrelated. They come from the poll loop at line 1075 that waits for the object to appear, one line per second until it does, and Chainsaw renders stdout and stderr as separate blocks, so they read as though they came after the condition met they actually preceded. The same three lines appear in the suite that passed.

There is a second defect underneath, and it cannot be assessed until this one is fixed: whether the 5-minute budget on the MachineDeployment gate is right. That budget was already raised once from 1 minute, for the reason the comment above it gives: post-Talos the bootstrap config gates on secrets that only exist after the Talos PKI and the apiserver ClusterIP do, and a cold sandbox pushes first scale-up out. Whether five minutes is now too tight cannot be measured while a timeout there is indistinguishable in the log from any other failure.