hack/run-kubernetes-node-join_test.bats guards the diagnostics phase against overrunning the Chainsaw op with an inequality: budget + largest collector cost + snapshot <= op - bringup. It holds today by ten seconds.
The terms, as the guard reads them: budget is parsed out of COZY_DIAG_PHASE_BUDGET_DEFAULT in the library and is 480; snapshot is parsed out of the timeout -k 30 360 crust-gather line and comes to 390 once the grace is added; largest is 620 and bringup is 1500, and those two are literals in the test. The window is 3000 - bringup, so 1500. That gives 480 + 620 + 390 = 1490 against 1500.
Ten seconds is not slack, it is the absence of it. Any one-second increase in any term fails the guard: a collector whose cost is re-measured, a crust-gather timeout raised by a second, a phase budget nudged up. The guard is doing its job in that case, since the sum genuinely would not fit, but the failure arrives as a red test in a file most people touching the diagnostics block have no reason to read, at a distance from the change that caused it.
The composition is what makes it worth filing rather than watching. Two of the three left-hand terms are derived from the code they describe and will follow it; largest and bringup are hand-written numbers that will not. largest is documented in the test as the heaviest collector's cost at the pool's minimum size, which is a deliberate under-estimate rather than a ceiling, and bringup is described in the test's own comment as an observed figure and explicitly not a ceiling. So the inequality is tight against two numbers that are already known to be optimistic, and the first person to add a collector to the phase will meet it.
Two directions a fix could take, and they are not equivalent. Widening the margin by raising the op ceiling moves the same problem one collector further out, which is the reasoning the phase budget was introduced with in the first place, so it should not be undone here. Deriving largest from the collectors rather than restating it would make the left-hand side follow the code, at the cost of the test needing to know each collector's cost; that is more work and more honest. Whoever picks this up should also record what the ten seconds are meant to absorb, because at present nothing says whether the margin is deliberate or residual.
hack/run-kubernetes-node-join_test.batsguards the diagnostics phase against overrunning the Chainsaw op with an inequality:budget + largest collector cost + snapshot <= op - bringup. It holds today by ten seconds.The terms, as the guard reads them:
budgetis parsed out ofCOZY_DIAG_PHASE_BUDGET_DEFAULTin the library and is 480;snapshotis parsed out of thetimeout -k 30 360 crust-gatherline and comes to 390 once the grace is added;largestis 620 andbringupis 1500, and those two are literals in the test. The window is3000 - bringup, so 1500. That gives480 + 620 + 390 = 1490against 1500.Ten seconds is not slack, it is the absence of it. Any one-second increase in any term fails the guard: a collector whose cost is re-measured, a
crust-gathertimeout raised by a second, a phase budget nudged up. The guard is doing its job in that case, since the sum genuinely would not fit, but the failure arrives as a red test in a file most people touching the diagnostics block have no reason to read, at a distance from the change that caused it.The composition is what makes it worth filing rather than watching. Two of the three left-hand terms are derived from the code they describe and will follow it;
largestandbringupare hand-written numbers that will not.largestis documented in the test as the heaviest collector's cost at the pool's minimum size, which is a deliberate under-estimate rather than a ceiling, andbringupis described in the test's own comment as an observed figure and explicitly not a ceiling. So the inequality is tight against two numbers that are already known to be optimistic, and the first person to add a collector to the phase will meet it.Two directions a fix could take, and they are not equivalent. Widening the margin by raising the op ceiling moves the same problem one collector further out, which is the reasoning the phase budget was introduced with in the first place, so it should not be undone here. Deriving
largestfrom the collectors rather than restating it would make the left-hand side follow the code, at the cost of the test needing to know each collector's cost; that is more work and more honest. Whoever picks this up should also record what the ten seconds are meant to absorb, because at present nothing says whether the margin is deliberate or residual.