test(e2e): redis-tls op timeout is smaller than the sum of its read caps #4250

Describe the bug

The verify-tenant-ca-projection script step in hack/e2e-chainsaw/redis/chainsaw-test.yaml wraps every read in k() { timeout 30 kubectl "$@"; }, and there are more capped reads in the step than its operation ceiling can cover. Eleven call sites run before the topology check, one of them inside a three-iteration loop, so thirteen capped executions. At the cap that is 390s against a ceiling that was 3m.

Nothing fails today, and the reason is worth writing down: reads normally finish in milliseconds, and the step measured 3.0s end to end on run 34711755303. The sum only matters if reads start running slow without failing.

What makes that possible rather than theoretical is the shape of the calls. Eight of the eleven put the read in a pipeline:

keys=$(k -n "$NAMESPACE" get secret redis-secure.tenant-ca \
  -o go-template='...' | sort | tr '\n' ' ')
src=$(k -n "$NAMESPACE" get secret redis-secure.ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d)

The script runs set -eu with no pipefail, so a read that burns its full 30s and then fails hands its status to the tail of the pipe and the script carries on to the next one. Only the three guarded forms stop it. So a degraded apiserver can spend the ceiling across several reads rather than being cut off at the first.

When the ceiling goes first, Chainsaw kills the shell and the step reports signal: killed with none of the messages the script would have printed. docs/agents/e2e-testing.md covers this for catch scripts: size the operation timeout above the inner budgets and let an inner one fire.

Environment

• Cozystack: main
• In-tree e2e lane

To Reproduce

Read the step and add up the caps. To see it, slow the apiserver so reads succeed near the cap instead of failing.

Expected behaviour

An inner cap fires and names what it was reading.

Actual behaviour

The operation ceiling can fire first, and then the step says only that it was killed.

Additional context

I raised this step to 5m in the fix for #4238, because that change adds a bounded wait whose failure message is the whole point of it and I did not want the ceiling eating it. That helps the margin and does not close the gap: 5m is still under the notional sum. The honest position is that the sum is notional and the real occupancy is three seconds, so this is about which instrument fires on a degraded run, not about a step that is too slow.

I have not looked at the other suites, so the question rather than the claim: do other script steps share this shape, and is the guidance already in the e2e doc worth turning into something mechanical?

The pipeline half of this overlaps #4246, which covers the same swallowed status from the angle of which cause gets reported. The budget consequence is separate, which is why it is here.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs