Skip to content

e2e: 50 kubectl reads inside the armed EXIT-handler window carry no time bound #4111

Description

The kubernetes suite arms an EXIT handler that captures a crust-gather snapshot of the tenant cluster. Inside that armed window most kubectl reads carry no time bound at all. A wedged apiserver there does not just fail the test: it spends the Chainsaw op budget, and the snapshot the handler exists to take is lost rather than truncated.

Counted on fb9b8c57b, in the window between trap '_tenant_failure_on_exit' EXIT and trap - EXIT (lines 5357-6234 at that head), excluding comments and calls made through a bounding wrapper, with line continuations joined before anything is discarded. 67 bare calls in the window:

category count
--request-timeout present 0
wrapped in timeout N 5
--timeout= wait deadline only, request itself unbounded 15
no time bound of any kind 47

Nine further reads are excluded because they go through a wrapper that bounds them (cozy_diag_read, _cozy_guard_kubectl, kubectl_wait_retry).

The 15 in the third row are listed separately on purpose. A kubectl wait --timeout= returns on its own deadline, so folding those into the total would overstate it by that much.

To recount: locate the window by those two trap lines rather than by line numbers, since the file moves. Join continuations into logical lines first, then drop comments and wrapper-mediated calls. The joining order matters and is the correction below.

Correction to the first version of this issue. It reported 50 unbounded reads and named get hr, describe hr and get events at lines 6140, 6142 and 6144 as the clearest candidates to bound. Both were wrong, from the same defect in the counting script: it tested a physical line for the word kubectl before joining continuations, so a two-line cozy_diag_read 'label' \ followed by the kubectl call lost its wrapper: the wrapper line was discarded as not containing kubectl, and its continuation was counted as a bare call. Those three calls are wrapped in cozy_diag_read, which bounds them with timeout -k. They were already bounded, and recommending they be bounded was the opposite of useful. The corrected count is 47, and the --request-timeout row falls from 2 to 0 for the same reason.

What this does not claim: that any of the 47 exceeds the budget it would need. Nobody has shown that, and nobody has shown otherwise — the premise is unchecked, not violated. It also does not say which to bound first; ranking needs per-call wall-clock from real runs, which nobody has collected.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Sep 7, 2026
  2. added
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  3. lexfrei commented on Oct 2, 2026

    @lexfrei
    ContributorAuthor

    Two more of the same: the LINSTOR pool helpers in hack/e2e-chainsaw/_lib/run-kubernetes.sh (the baseline capture and wait used in the kubernetes suites' cleanup) run kubectl exec into the LINSTOR controller with no time bound, so a hung exec holds the cleanup until the op timeout instead of failing with its own message. Found while writing the computeplane suite (#4713).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions