Skip to content

e2e: dataplane capture can spend its whole outer budget on one hung kubectl get, and truncation is silent #3642

Closed

Description

Summary

hack/e2e-capture-dataplane.sh carries six kubectl get calls with no per-call bound (lines 261, 271, 387, 425, 427, 614 at 605030b), while every one of its 27 kubectl exec calls goes through a bounded wrapper and three of its other kubectl get calls carry an inline timeout 20 (lines 596, 635, 644). The script's own header comment (lines 242-246) is explicit that wall-clock is bounded only by the caller's envelope: 600s from the cozytest.sh EXIT trap, 300s from the Chainsaw global catch.

Why it matters

Against an apiserver that hangs rather than refuses (a runner whose kubeconfig points at a dead endpoint, an apiserver mid-crash), the very first unbounded call, kubectl get pods -A on line 261, consumes the entire envelope before any capture is written. From the Chainsaw catch the result is worse than empty: the capture is cut off by the outer timeout -k, and the script writes no note about being cut short, so a truncated capture is indistinguishable from a complete one that found nothing. Its sibling e2e-capture-previous-logs.sh already solves both halves: it self-caps its listing (PREVLOG_LIST_TIMEOUT=28, line 187) and logs "the read was cut off by ..." when a bound fires (line 381).

This is the same class as the collector-bounding work that recently landed for the report collector and the tenant cleanup waits; this script is the remaining member.

Fix shape

Give the six calls the same inline bound the script already uses elsewhere (timeout 20, with the existing grace pattern), sized so the sum of worst cases stays inside the 300s Chainsaw envelope, and emit a cut-off note in the sibling's discipline whenever a bound fires so a truncated capture identifies itself.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Aug 8, 2026
  2. lexfrei commented on Aug 8, 2026

    @lexfrei
    ContributorAuthor

    Correcting the fix shape in the description: "sized so the sum of worst cases stays inside the 300s Chainsaw envelope" is not achievable and was never the script's design. Measured at 605030b: the per-pod loop alone carries about 250s of per-call bounds per iteration, and the loop runs up to MAX_PODS=12, so the worst-case sum of the already-bounded calls is on the order of 50 minutes. The script's own header comment (lines 242-246) states the intended contract: the caps bound WORK, and wall-clock is held by the caller's outer timeout -k backstop.

    The right goal for the six unbounded calls is narrower: no single read may consume the caller's envelope. Today a hanging apiserver spends the whole 300s/600s budget on the first kubectl get pods -A and the capture writes nothing at all; with per-call bounds the script keeps making progress and leaves a partial capture that identifies itself via the cut-off note. Partial and honest instead of empty, not "everything fits in 300s".

  3. lexfrei commented on Sep 23, 2026

    @lexfrei
    ContributorAuthor

    Fixed by #3659. Every read in the dataplane capture now has its own bound, and a cut-off read is reported instead of silently truncated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugtriage/not-reproducibleIndicates an issue can not be reproduced as described

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions