Repository navigation
testing: cozytest.sh arms cluster captures on unit-suite failures, turning a fast red into ~15 minutes of dead waits #3626
Description
Activity
Correcting the cost estimate in the description, measured with a kubectl stub that hangs instead of refusing.
e2e-capture-previous-logs.shnever reaches the 300s outer cap: it self-limits its pod listing (PREVLOG_LIST_TIMEOUT=28) and exits 0 after about 28s.e2e-capture-dataplane.shdoes run its full 600s, killed by the outer bound on its first unboundedkubectl get pods -A. So the realistic worst case on a runner without a cluster is about 10 minutes, not 15, plus the 360s crust-gather leg where that tool is installed.The measurement also explains why this survived so long: the cost depends on whether kubectl fails or hangs. On a laptop with no kubeconfig context it refuses instantly and the whole trap costs about a second, so the defect is invisible locally. It only bites where the context points at an endpoint that accepts the connection and stalls, which is exactly the CI-runner shape.
The dataplane script's unbounded calls are now tracked separately in #3642.
Fixed by #3645.
hack/cozytest.shonly arms the cluster captures for e2e suites now, so a failing unit suite doesn't walk into their timeouts.
Summary
hack/cozytest.sharms its cluster-capture EXIT trap for everyhack/*.batsfile it runs, including purely local unit suites. The gates inside the trap arecommand -v kubectland the capture script being executable (hack/cozytest.sh:110,:166), not "is there a cluster to talk to". On a runner that has a kubectl binary and no cluster, the first failing unit test therefore does not fail fast: the trap walks into the previous-logs capture under atimeout -k 30 300and the dataplane capture under atimeout -k 30 600, so a red that should cost seconds can cost up to ~15 minutes of dead waits inside a 30-minute job budget.Mechanism
The trap at
:177fires on the first failing test (the runner exits on first failure by design). Each capture is individually bounded, which is correct for the e2e context they were written for, but the bound is the failure mode here:kubectlexists on the unit runners, every API call it makes hangs or fails slowly against nothing, and the timeouts serially add up. The captures read state that does not exist for a unit suite, so the cost buys an empty snapshot.Fix shape
Gate the trap on the context rather than on binary presence: arm the cluster captures only when an e2e marker is present (an env the e2e workflow sets, or a reachable API server probed with a short
--request-timeout), and let unit suites die with plain output. The per-capture bounds stay as they are for the e2e path.