The diagnostics phase budget introduced with the per-read bounds can be spent almost entirely on the cheap reads that precede the heavy collectors, leaving nothing for the collectors the phase exists to run.
The numbers are in hack/e2e-chainsaw/_lib/run-kubernetes.sh: COZY_DIAG_PHASE_BUDGET_DEFAULT=480, COZY_DIAG_READ_TIMEOUT_DEFAULT=20, COZY_DIAG_READ_GRACE_DEFAULT=5. Each bounded read therefore costs up to 25 seconds, and the node-join failure block issues about fifteen of them before the first heavy collector, plus an importer listing and a wedge check with their own bounds. At the ceiling that is roughly 435 to 460 seconds of the 480, so the serial console (~350s), the guest Talos capture and the image-cache diagnosis are all declined.
Two things keep this from being alarming on every run, and both are worth stating so the number is not read as larger than it is. The ceiling is only reached when the reads genuinely hit their deadlines, which means a wedged apiserver. In the ordinary failure shape, where the apiserver answers and the worker simply never boots, those reads return in milliseconds and the budget is barely touched. And the budget is a ceiling on what may start, not a prediction of a run.
But the case where the ceiling is reached is exactly the case the diagnostics exist for. A wedged apiserver is when the cheap reads are least informative and the guest-side captures are most. Spending the phase on the former and declining the latter inverts the intent.
There is a second, smaller problem in the gate itself. cozy_diag_phase_has_time admits a collector if the deadline has not passed yet:
cozy_diag_phase_has_time() {
[ "${_COZY_DIAG_PHASE_DEADLINE}" -ne 0 ] || return 0
[ "$(date +%s)" -ge "${_COZY_DIAG_PHASE_DEADLINE}" ] || return 0
...So a collector admitted one second before the deadline runs to completion and overruns the budget by its whole cost. The gate bounds when work may begin, not what fits, which means the budget is advisory for the last admitted collector and absolute for everyone after it.
Finally, the comment in both chainsaw-test.yaml files states that the reads ahead of the heavy collectors fit inside the budget "with room left". At the ceiling that is 435 to 460 of 480. The sentence describes the ordinary case while sitting next to an inventory whose purpose is to size the pathological one, and it is the sentence the next person adding a collector will read.
Directions, in the order they seem worth considering. Give the cheap reads their own sub-budget so they cannot consume the phase. Or order the phase so the irreplaceable captures run before the reads that merely describe state. Or make the gate reason about cost rather than start time, which requires each collector to declare a bound the gate can compare against the remainder.
The diagnostics phase budget introduced with the per-read bounds can be spent almost entirely on the cheap reads that precede the heavy collectors, leaving nothing for the collectors the phase exists to run.
The numbers are in
hack/e2e-chainsaw/_lib/run-kubernetes.sh:COZY_DIAG_PHASE_BUDGET_DEFAULT=480,COZY_DIAG_READ_TIMEOUT_DEFAULT=20,COZY_DIAG_READ_GRACE_DEFAULT=5. Each bounded read therefore costs up to 25 seconds, and the node-join failure block issues about fifteen of them before the first heavy collector, plus an importer listing and a wedge check with their own bounds. At the ceiling that is roughly 435 to 460 seconds of the 480, so the serial console (~350s), the guest Talos capture and the image-cache diagnosis are all declined.Two things keep this from being alarming on every run, and both are worth stating so the number is not read as larger than it is. The ceiling is only reached when the reads genuinely hit their deadlines, which means a wedged apiserver. In the ordinary failure shape, where the apiserver answers and the worker simply never boots, those reads return in milliseconds and the budget is barely touched. And the budget is a ceiling on what may start, not a prediction of a run.
But the case where the ceiling is reached is exactly the case the diagnostics exist for. A wedged apiserver is when the cheap reads are least informative and the guest-side captures are most. Spending the phase on the former and declining the latter inverts the intent.
There is a second, smaller problem in the gate itself.
cozy_diag_phase_has_timeadmits a collector if the deadline has not passed yet:So a collector admitted one second before the deadline runs to completion and overruns the budget by its whole cost. The gate bounds when work may begin, not what fits, which means the budget is advisory for the last admitted collector and absolute for everyone after it.
Finally, the comment in both
chainsaw-test.yamlfiles states that the reads ahead of the heavy collectors fit inside the budget "with room left". At the ceiling that is 435 to 460 of 480. The sentence describes the ordinary case while sitting next to an inventory whose purpose is to size the pathological one, and it is the sentence the next person adding a collector will read.Directions, in the order they seem worth considering. Give the cheap reads their own sub-budget so they cannot consume the phase. Or order the phase so the irreplaceable captures run before the reads that merely describe state. Or make the gate reason about cost rather than start time, which requires each collector to declare a bound the gate can compare against the remainder.