Repository navigation
e2e: the dataplane LB fan-out caps are measured in work, not wall-clock, and overrun both backstops #3671
Description
Activity
- added a commit that references this issue
on Aug 8, 2026 Scope update from the branch that adds a healthy-pod reference to the per-pod capture (PR #3617, not yet merged): the per-pod half of the budget question roughly doubles there. Measured against that branch:
capture_pod_dataplanecosts about 215s of bounded work per invocation, the reference adds a second invocation of the same function plus a node pod-list read (28s plus grace), so an affected pod costs on the order of 245s. AtMAX_PODS=12that is about 2900s of work against the same 600s and 300s backstops this issue is about.Worth recording alongside the LB numbers, because the two halves compound: the pod path runs before the LB section, so on a broadly degraded cluster the LB fan-out may never be reached at all. The
MAX_PODScomment that claimed the work tracks "#affected-nodes rather than #affected-pods" was weakened by that change and has been corrected on the branch itself; the cap decision belongs here.Correction to the scope update above, which described PR #3617 at an earlier commit and is now wrong in one respect. The reference capture is taken once per NODE and cached, not once per affected pod, so the ~245s it adds (a
capture_pod_dataplaneinvocation plus a node-scoped pod list) is paid K times for K nodes carrying affected pods, not 12 times atMAX_PODS=12. The per-pod leg of the worst case therefore stopped growing linearly in pods and now grows in nodes, like the node-global part already did.What that correction does NOT change is the substance here.
capture_pod_dataplanestill costs about 215s per invocation and still runs once per affected pod, since only the reference is shared. The breakdown, so the number can be checked rather than taken on faith: 135s of exec budget (25s cilium CT, 15s ip route get, 15s conntrack, 20s ovn-sbctl, 20s ovn-nbctl, 20s ovs-vsctl find, 20s ovs-vsctl get), two boundedkubectl getcalls at 20s each, and one 20sovs-vsctl find. The pod leg still runs before the LB section, so on a broadly degraded cluster the LB fan-out may not be reached at all, and the gap against the 300s Chainsaw envelope is unchanged — the reference simply stopped doubling it.The
MAX_PODScomment claiming the work tracks "#affected-nodes rather than #affected-pods" is closer to true after that change than it was when the comment was written, but it describes the reference, notcapture_pod_dataplaneitself, which is the larger term. The cap decision this issue is about stands either way.Correcting the per-invocation number in my two comments above:
capture_pod_dataplanecosts 175s of bounded work, not 215s. Verified line by line againstfix/e2e-capture-address-attribution, listing the matches instead of summing a search:- seven
timeoutexec bounds: 25s (cilium CT), 15s (ip route get), 15s (conntrack), 20s (ovn-sbctl), 20s (ovn-nbctl), 20s (ovs-vsctl find), 20s (ovs-vsctl get) = 135s - two bounded
kubectl getcalls (Ready conditions, events) atDP_READ_TIMEOUT20s = 40s
That is 175s nominal, or 179s counting the 2s kill grace on the two calls that carry
timeout -k(the exec bounds use baretimeout, which has no grace). The 215s figure came from counting occurrences of the bound token rather than bounded calls:$DP_BOUNDappears four times in the function, twice as a command prefix and twice as an argument to the outcome helper inside|| echo "(reading ... )", and the two argument uses were counted as calls. The published breakdown also listedovs-vsctl findtwice, once inside the 135s exec group and again as a separate term, so it did not add up to its own total either — 135 + 40 + 20 is 195, not 215. Both are my error; the earlier 175s estimate in the review thread was right.Derived numbers that shift with it: the per-node reference costs 175s plus the ~30s node-scoped pod list = 205s, not 245s, and the "~2900s at
MAX_PODS=12" in the first scope update was computed from the inflated figure and is correspondingly high.None of this moves the issue. 175s per affected pod still runs once per affected pod, the pod leg still precedes the LB section, and the gap against the 300s Chainsaw envelope and the 600s backstop is what the issue is about. The point of the correction is that a number in a tracking issue gets reused: a wrong figure that looks checkable is worse than an honest estimate, because the next person will build a cap decision on it instead of re-deriving it.
- seven
The per-pod side of this now overruns the wider backstop too, not just the Chainsaw one. Measured on
fix/e2e-capture-address-attribution, where the arithmetic also landed in theMAX_PODSnote so it can be recomputed instead of trusted: a per-pod capture bounds at 179s (135s of execs plus two 20s reads with their 2s graces), a node capture at 250s, and the healthy-pod baseline adds 209s per affected node (its own capture plus the 30s node-scoped list). One affected pod on one node therefore bounds at 638s, against the 600s thecozytest.shtrap allows and the 300s the Chainsaw catch shares with its snapshot leg.So the smallest possible case no longer fits the roomiest envelope we have. Everything above this comment was about the 300s one.
These are ceilings that need every exec to hang at once, not an expectation, and both callers do report a capture their backstop cut short, so the failure is loud rather than silent. What the numbers settle is the shape of the cap itself: it counts pods while the ceiling is measured in seconds, and at this size the two stop meeting. Raising
MAX_PODSbuys nothing either envelope can pay for, and the open question in this issue (express the cap in wall-clock rather than in captured-pod count) applies to the pod leg exactly as it does to the LB one.Superseding my earlier figures here: 175s per invocation was correct for the code as it stood then; 179s is the same call counted with the kill graces the two bounded reads carry.
Correcting the number in my previous comment: the minimum case bounds at 676s, not 638s. Recomputed from the source line by line rather than from the note.
The node capture is 222s of exec bounds (25, 12, 25, 25, 25, 20, 15, 15, 15, 15, 15, 15) plus its three
pod_on_nodelookups at 22s each, so 288s. The earlier 250s came from a search that counted two numbers written in a comment describing the bounds, and counted atimeout 8nested inside atimeout 12as a second wait rather than the same one. Separately the three lookups were not counted at all, while the per-pod figure did count its two bounded reads, so the two terms were built by different rules.Applying one rule to all three terms (each call's outer bound plus its bounded lookups, a nested inner timeout counted once) gives 288 + 179 + 209 = 676s, against the 600s the
cozytest.shtrap allows and the 300s the Chainsaw catch shares with its snapshot leg. The note inMAX_PODSnow states that counting rule explicitly, so the next person can recompute instead of trusting the total.The conclusion holds and gets slightly stronger. The smallest possible case does not fit the roomiest envelope; only the size of the gap moved.
One more input for the cap decision, about collection order rather than size. On the branch that adds the healthy-pod reference, the reference for a node is captured immediately after the first affected pod on that node. So when a backstop truncates the run, the envelope has already been spent on a control capture while affected pods 2..N were never captured at all. Truncation eats the primary evidence before it eats the optional kind.
That matters more here than it would elsewhere, because the numbers above say truncation is the normal case for this collector rather than a corner: the smallest possible case bounds past the wider backstop. Whatever the cap ends up being expressed in, the order the section collects in decides what survives a cut, and right now it favours the control.
Deliberately not changed on that branch. Moving the reference pass after the affected-pod loop is a second walk over the nodes and a new node in a part of the code that has already needed three corrections, and it would not remove the loss, only redistribute it. The complete answer is the one this issue is already about (a wall-clock deadline for the section instead of a captured-count cap), and the ordering question should be settled together with it rather than patched ahead of it.
Observed in CI rather than computed, which is worth recording because everything above this is arithmetic. On a run of the
kubernetes-previoussuite, the failing step ended at 01:00:14, the catch leg started there, and at 01:10:34 it ended withsignal: killed. Ten minutes to the second: the 600s backstop fired and killed the collector mid-capture.The suite failed for an unrelated and already-known reason (
timed out waiting for the condition on helmreleases/...-csi), so this is not a case of the collector causing the failure. It is the ordinary path: a suite fails, the capture leg runs, and the backstop cuts it off. The same run also shows the teardown leg reportingtenant teardown did not drain within 300swith three PVCs still bound, so both envelopes were exhausted on the same failure.So the ceilings in this issue are not a worst case that needs every exec to hang at once. Something in that leg already spends the whole 600s on a normal failing run, and whatever it had not written by 01:10:34 is not in the artifact.
Worth pointing at when deciding the cap: the question is not only whether the numbers fit, but which parts of the capture survive the cut, and right now nothing chooses that ordering deliberately.
Updating the figure again, and this time not because the earlier one was measured wrong: the code changed underneath it. The baseline no longer performs a read of its own.
The branch that adds the healthy-pod reference now selects it from the cluster-wide pod snapshot that the collector already takes, instead of a second, node-scoped list.
capture_pod_referencemakes nokubectlcalls at all, and the three pod lookups inside the baseline capture are memo hits, becausecapture_noderuns before it for the same node. So the baseline costs one capture, not a capture plus a list.That makes the minimum case 288 + 179 + 179 = 646s, against the 600s the
cozytest.shtrap allows and the 300s the Chainsaw catch shares with its snapshot leg. The previous 676s was correct for the code as it stood when it was posted, and the 30s difference is the list that no longer happens.The conclusion is unchanged for the third time in a row, which is worth stating plainly: every correction to this number so far has moved it while leaving the finding intact. The smallest possible case does not fit the roomiest envelope, and the cap counts pods while the ceiling is measured in seconds.
The arithmetic now lives in the
MAX_PODSnote in the script with its counting rule spelled out, so the next person can recompute rather than trust any of the figures in this thread.One condition on the 646s figure above, and one comparison that puts it in context.
646s assumes the per-node memo is available. When
mktemp -dfails, the memo is empty,pod_memo_getreturns immediately, and the three pod lookups inside the baseline capture stop being hits. Measured with a stub rather than reasoned: a working TMPDIR produces 7 field-selector calls, a read-only one produces 13. In that state the minimum case bounds at 288 + 245 + 245 = 778s, where 245 is the 179s capture plus three 22s lookups. The collector degrades correctly there and still reports, so this is a cost, not a failure, but the figure should not be read as unconditional.For context on merge-base: the same single-affected-pod case bounded at 437s, which fit inside the wider of the two backstops. The healthy-pod baseline is the one term that moves this case past both. Worth stating plainly because the pod section runs before the LoadBalancer datapath capture, so a cut here costs the whole LB capture, and the LB capture exists for a failure mode where no NotReady pods are present at all.
Worth recording that the 300s envelope was already out of reach before any of the changes discussed here. On merge-base the single-affected-pod case bounds at 437s: a node capture plus one per-pod capture, with no baseline involved. So the Chainsaw catch has never been able to contain a worst case of this collector, and the recent movement is about the 600s backstop, which merge-base did fit and the current tree does not.
That matters for how the cap decision gets framed. Treating both envelopes as constraints to satisfy sets a target that was never met; treating the 300s one as a known overrun and the 600s one as the line that just moved gives an accurate picture of what changed and what to aim at.
The branch adding the healthy-pod baseline no longer moves the single-affected-pod case past the wider backstop. The baseline is now captured at a reduced scope: route and conntrack only, dropping the cilium CT dump, the OVN Port_Binding and LSP reads and the two
ovs-vsctlcalls, which is 105s of the 135s exec budget. Those four describe how a pod is programmed, and the baseline pod is programmed correctly by definition, so they were never the comparison the ticket asks for.That puts the minimum case at 288 + 179 + 74 = 541s, where 74 is 30s of execs plus the same two 22s reads. Inside the 600s the
cozytest.shtrap allows, still past the 300s the Chainsaw catch shares with its snapshot leg, and 673s if the per-node memo is unavailable.So the specific regression this thread tracked is closed: the case that fit on merge-base at 437s fits again. What the issue is about does not change with it. The cap still counts pods while the ceiling is measured in seconds, the 300s envelope is still unreachable and was unreachable before any of this, and the pod section still runs before the LoadBalancer capture, so a cut still costs the LB evidence entirely.
Correcting the comment above: the reduced-scope baseline does not bring the single-affected-pod case back inside the 600s backstop. 541s counted only the three terms named there and left out the preamble the script runs before reaching any of them: the cluster-wide pod list at 30s, the
ovn-centrallookup at 22s, and theovn-sbctl lflow-listat 30s. That is 82s, and it applies to every run.Recounted line by line:
head, reduced-scope baseline : 82 + 288 + 179 + 74 = 623s head, full-scope baseline : 82 + 288 + 179 + 179 = 728s merge-base : 82 + 258 + 179 = 519sSo the reduced scope removes 105s of a 728s worst case rather than closing the gap: 623s is still past 600s. And the omission changed the conclusion rather than the wording, which is the part worth flagging. On merge-base the same preamble gives 519s, comfortably inside; after the change it is 623s, outside. The regression the earlier comment claimed to have closed is real and still open.
The narrower question of whether the baseline should carry conditions and events at all is being looked at, since the pod's Ready state is already known from the snapshot it was selected by. That is a further 44s if it goes.
- added a commit that references this issue
on Aug 9, 2026 - added a commit that references this issue
on Oct 5, 2026
Summary
In
hack/e2e-capture-dataplane.shthe fan-out caps are measured in units of WORK while the backstops that contain them are measured in wall-clock, and the worst case exceeds both by several multiples. Per unreachable LoadBalancer the heavy capture costs roughly 282s:capture_lb_nodecarries 25+15+15+25+20 = 100s of per-call bounds and runs for two nodes (announcer and endpoint) = 200s, plusovs_iface_forat 20s, plus a tcpdump section of about 26s (two background execs of 15s in parallel, then a 2s sleep and two repeat probes of 12s), plus the gating probes at 3x12 = 36s. WithMAX_LBS=6(line 303) that is on the order of 1700s, against a 600s backstop fromcozytest.shand a 300s one from the Chainsaw catch, where the 390s tenant snapshot also has to fit. The pod path withMAX_PODS=12(line 287) runs before any of that.The script's own
lb_budget_okcomment states the design honestly ("bounds WORK, not wall-clock"), so this is a known shape rather than an oversight; what is missing is any check that the shape still fits the envelopes it runs in.Why it matters
The caps cannot prevent the backstop from firing on a broadly degraded cluster, which is exactly the state this collector exists for. Until recently a cut-off capture was silent, so the overrun looked like a small artifact; with the collector now reporting an INCOMPLETE cut-off, the mismatch becomes visible on every such run and will read as a collector bug rather than a budget question.
Fix shape
Two questions to settle together. First, add the budget test the sibling collector already has (
hack/capture-previous-logs.bats, "the inner read budgets stay under the backstop the callers wrap this in");hack/capture-dataplane.batshas no equivalent, on the merge base either. Second, decide whetherMAX_LBS=6is defensible under the 300s Chainsaw envelope or whether the cap should be expressed in wall-clock (a deadline for the LB section) rather than in captured-LB count.