You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
e2e: dataplane capture can spend its whole outer budget on one hung kubectl get, and truncation is silent #3642
hack/e2e-capture-dataplane.sh carries six kubectl get calls with no per-call bound (lines 261, 271, 387, 425, 427, 614 at 605030b), while every one of its 27 kubectl exec calls goes through a bounded wrapper and three of its other kubectl get calls carry an inline timeout 20 (lines 596, 635, 644). The script's own header comment (lines 242-246) is explicit that wall-clock is bounded only by the caller's envelope: 600s from the cozytest.sh EXIT trap, 300s from the Chainsaw global catch.
Why it matters
Against an apiserver that hangs rather than refuses (a runner whose kubeconfig points at a dead endpoint, an apiserver mid-crash), the very first unbounded call, kubectl get pods -A on line 261, consumes the entire envelope before any capture is written. From the Chainsaw catch the result is worse than empty: the capture is cut off by the outer timeout -k, and the script writes no note about being cut short, so a truncated capture is indistinguishable from a complete one that found nothing. Its sibling e2e-capture-previous-logs.sh already solves both halves: it self-caps its listing (PREVLOG_LIST_TIMEOUT=28, line 187) and logs "the read was cut off by ..." when a bound fires (line 381).
This is the same class as the collector-bounding work that recently landed for the report collector and the tenant cleanup waits; this script is the remaining member.
Fix shape
Give the six calls the same inline bound the script already uses elsewhere (timeout 20, with the existing grace pattern), sized so the sum of worst cases stays inside the 300s Chainsaw envelope, and emit a cut-off note in the sibling's discipline whenever a bound fires so a truncated capture identifies itself.
Correcting the fix shape in the description: "sized so the sum of worst cases stays inside the 300s Chainsaw envelope" is not achievable and was never the script's design. Measured at 605030b: the per-pod loop alone carries about 250s of per-call bounds per iteration, and the loop runs up to MAX_PODS=12, so the worst-case sum of the already-bounded calls is on the order of 50 minutes. The script's own header comment (lines 242-246) states the intended contract: the caps bound WORK, and wall-clock is held by the caller's outer timeout -k backstop.
The right goal for the six unbounded calls is narrower: no single read may consume the caller's envelope. Today a hanging apiserver spends the whole 300s/600s budget on the first kubectl get pods -A and the capture writes nothing at all; with per-call bounds the script keeps making progress and leaves a partial capture that identifies itself via the cut-off note. Partial and honest instead of empty, not "everything fits in 300s".
Activity
lexfrei commentedon Aug 8, 2026
Correcting the fix shape in the description: "sized so the sum of worst cases stays inside the 300s Chainsaw envelope" is not achievable and was never the script's design. Measured at 605030b: the per-pod loop alone carries about 250s of per-call bounds per iteration, and the loop runs up to
MAX_PODS=12, so the worst-case sum of the already-bounded calls is on the order of 50 minutes. The script's own header comment (lines 242-246) states the intended contract: the caps bound WORK, and wall-clock is held by the caller's outertimeout -kbackstop.The right goal for the six unbounded calls is narrower: no single read may consume the caller's envelope. Today a hanging apiserver spends the whole 300s/600s budget on the first
kubectl get pods -Aand the capture writes nothing at all; with per-call bounds the script keeps making progress and leaves a partial capture that identifies itself via the cut-off note. Partial and honest instead of empty, not "everything fits in 300s".fix(tests): bound the e2e data-plane capture's reads and stop inferri…
docs(tests): correct five stale claims in the e2e capture tooling (#3662
lexfrei commentedon Sep 23, 2026
Fixed by #3659. Every read in the dataplane capture now has its own bound, and a cut-off read is reported instead of silently truncated.