Skip to content

test(e2e): every bounded diagnostic read pairs equal wall-clock and request timeouts, so the request deadline never fires #3690

Open

Description

Every bounded read in the node-join diagnostics block wraps kubectl in a wall-clock timeout and passes --request-timeout at the same number of seconds. Both halves come from one knob: cozy_diag_read spends COZY_DIAG_READ_TIMEOUT as its wall clock (hack/e2e-chainsaw/_lib/run-kubernetes.sh:1435) and the block composes the flag from the same variable (:1493, local request_timeout="--request-timeout=${COZY_DIAG_READ_TIMEOUT}s"), which is then handed to all 18 cozy_diag_read call sites. The importer listing (:1568) and the worker CPU throttling capture (:955, :1032) call timeout directly with the same pair.

Equal is not merely redundant, it is ordered. The wall clock starts when the process starts; kubectl's request deadline starts only once the client has loaded its config and opened the connection, which is strictly later. So the outer kill always arrives first and the inner flag never gets to expire -- it cannot report a cause, and the read ends as a bare exit 124 with no text.

This is measured rather than reasoned. cozystack/cozystack#3677 drove the same construction against a stub apiserver in three failure modes and found the third is the one that matters: a refused port answers immediately and names the refusal, a stall mid-TLS is ended by kubectl's own handshake deadline and still produces text, and a blackholed address -- SYN dropped with no RST, which is what a wedged node or a NetworkPolicy looks like -- produces nothing at all until some deadline expires. Its controlled A/B against the same target: 30 over 30 gives exit 124 and zero bytes; 40 over 30 gives exit 1 at the thirty-second mark and a message naming the cause.

That PR fixed the pairing in the code it owned and deliberately did not touch this block, saying so in its own body: the diagnostic reads here run off a validated, overridable pair, and wiring its lookup to those knobs would let a test that turns the dump budget down silently change which image factory a suite pulls from. The reasoning for leaving it alone was about coupling, not about the pairing being correct here.

Two things a fix has to keep in view. The single value is load-bearing in one direction: the comment at :1488-1491 says it exists so the two halves cannot disagree -- a zero assigned after sourcing would otherwise produce timeout -k 5 20 over --request-timeout=0s. Whatever replaces it needs to keep that coupling while separating the numbers, most simply by deriving the wall clock from the request budget plus a fixed headroom rather than by introducing a second independent knob. And the flag is not decoration everywhere: three reads carry a command -v timeout fallback, and in that branch --request-timeout is the only bound there is, so it must not be dropped in the name of removing a redundant one.

Cost is bounded and worth naming: separating the two adds the headroom, once per read that actually hangs, on a path where the cluster is already unresponsive. Nothing changes on a healthy run, where reads return in milliseconds.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions