fix(e2e): bound the dataplane capture collector by wall clock - #4640

…bles

POSIX sh has no local scope, so every assignment in a function lands in
the caller. capture_node wrote `node`, the variable of the loop that
calls it, and capture_lb_node wrote the LB loop's _lbip, _np, _epip and
_of. Both stayed harmless only because each caller passed what it
already held; any other argument would have made the rest of the
iteration read another node's pods and file them under the right name.

Both functions now use their own prefix, like the newer helpers in the
file, and a static test fails when a function assigns a name that a
`read -r` loop or another scope also owns.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

`cilium-dbg bpf lb list` ran only inside the LoadBalancer path, which is
entered only when a LoadBalancer IP fails its probe. The map describes
every service translation on the node, so a pod that cannot reach a
ClusterIP left no record of which backend the VIP resolved to or
whether a backend that stopped answering was still in the map.

It now runs unfiltered in the per-node capture, which runs for the node
of each affected pod the collector reaches.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Three collector tests shrank both per-read bounds to one second to make
the read under test fail fast. The same bounds also cover the reads that
must answer, so on a loaded machine the service, endpointslice or pod
list is cut instead, the collector correctly takes another branch, and
the test fails as if the code under test had regressed.

A `timeout` shim on PATH now cuts only the reads whose argv the test
names and runs the rest unbounded against the instant stub, so the cut
is the input rather than a race. A cut read that bypasses `timeout`
still reaches the stub's sleep and runs into the test's own backstop. The
tests also assert the collector finished on its own and the decision
line it logs, not only that a file exists.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The fan-out caps counted pods and LoadBalancers while both callers'
backstops count seconds, and a single affected pod's worst case already
ran past the 300s Chainsaw one. No count could be made to fit, and a
cut-off run lost whatever came last, which was always the LoadBalancer
capture.

Every kubectl call now goes through dp_run, which clips the call's
bound to what is left of a run-wide budget sized under the smaller
backstop and starts nothing once it is spent. The pod section stops a
reserve short of the end so the LoadBalancer section always gets time
to start. A read cut by a clipped bound quotes the bound it ran under,
a read never started says so instead of naming a timeout, and a section
that runs out of budget names what it did not capture.

Tests pin the budget against both callers' backstops, the behaviour on
a cluster where every read hangs, and that no kubectl call bypasses
dp_run, listing any that does by line.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

A call dp_run refuses for a spent budget returned silently, and every
exec block swallows its status, so a capture the budget ran out in the
middle of kept only section headers. A header with nothing under it
reads as a command that found nothing, and the notes said nothing
either, because the loops only decline whole pods and LBs.

The refusal now goes to stderr, which lands in the block it emptied,
and capture-notes.txt gets one line per run. The OVS interface lookup
also stops calling a failed, cut or refused lookup an absent interface.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The per-node lookup memo keeps a cutoff so a hung lookup is not paid
for twice. Once bounds are clipped to the budget, a lookup can be cut
after a second or two because the pod section's deadline arrived, and
caching that made the LoadBalancer section report the component as
unknown for the node while its reserve still had time to ask.

A cutoff is now cached only when the lookup ran under its full bound.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The per-LB capture and its two tcpdump files joined namespace and
Service name with a hyphen, which both may contain, so a-b/c and a/b-c
wrote the same files, and each run truncated them while the notes
beside them accumulate. They now use the separator and the run marker
the per-pod files use.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The pod section stops a reserve short of the end of the budget and the
LoadBalancer section keeps starting calls after it, but a refusal was
reported as the whole run's budget running out, which the LB capture
written below it contradicts. The note was also deduplicated once per
run, so the LB section running out later left no line of its own.

Refusals now name their section, and the note is written once per
section. Both budget knobs are also read as decimal: shell arithmetic
took a leading zero for octal, so 08 aborted the run before it wrote
a note and 010 meant 8.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

…ackstop

The collector's default budget is sized under the 300s Chainsaw
backstop. The cozytest.sh caller allows 600s, and on the default it
would give up more than half of that window, so it passes a budget
matching its own backstop. The budget test reads each caller's
override and fails when a budget leaves more of its backstop unused
than the backstop's kill grace.

The Chainsaw call-site comment still described the fan-out caps as the
bound, and the e2e testing guide described an EndpointSlice exception
to the notes contract that the code does not have; both now describe
what the collector does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

…topped

The budget can run out inside one capture rather than between them. A
node lookup refused by dp_run was then written up as a lookup that did
not answer, an LB probe whose cni-server lookup was refused as the same,
and a probe exec refused, or cut by a bound the deadline had shortened,
as a node with no cni-server or no probe client. Each names a cause that
never happened.

The lookup's own status now reaches its callers, the node capture says
the lookup was not asked, and the probe reports a "spent" outcome that
the decision weighs like an unrun probe and the reason line attributes
to the section's budget. A reserve at least as large as the budget is
also named, since the pod section then starts no call at all.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

A few reads that go through dp_run still failed or fell back without a
trace. A speaker log read that failed looked like a
speaker that never announced the address; it now writes a note. The
endpoint tcpdump's fallback to the geneve interface is stated in the LB
capture. The node capture now carries the run separator the pod and LB
files use, through one helper.

Two budget interactions named a cause that never happened. A per-read
timeout knob of zero made every read look refused by the budget, and a
value timeout cannot parse failed every read with 125, reported as
kubectl failing; both knobs are now validated like the budget ones. An
LB probe whose cni-server lookup the deadline cut was reported as a
second, unanswered lookup; it now counts as stopped by the budget.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Once bounds are clipped to the budget, a cutoff has two meanings: the
call's own bound fired, or the deadline shortened it. Node lookups cut
the second way were still written up as lookups that did not answer, a
cause about the cluster that nothing observed. And the section note was
written only when a call was refused, so when the deadline cut a
section's last call the truncated capture carried no word about the
budget at all.

A lookup cut under a bound the deadline shortened now names the budget
in the node capture, the endpoint interface lookup and the LB probe,
decided from the bound it ran under rather than from the clock when its
status is read. A call the deadline cuts also writes the section
note, as a refusal does. The heavy-capture line also stops claiming
every probe failed when some of them were stopped by the budget.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

…early

The pod walk can now also stop when the pod section's share of the
budget runs out, as it already could at the pod cap. Both exits name
what they left behind, and the line after them still announced the
capture as complete. It now says the walk stopped early.

The helper comment about the run separator also named every capture
file, while the OVN dump and the speaker logs are still replaced on
each run; it now lists the files it covers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The LoadBalancer walk can stop on the budget, as the pod walk can, and
still closed on "capture complete" right under the line naming what it
did not probe. Both walks also called themselves complete when they
ended on their own after the deadline had cut a call inside their last
item.

Each walk now closes as stopped early when it broke out of the loop,
and as cut short by the budget when this run's section note says a
call was cut or refused; the note's own text is the evidence, since the
clock read after the loop can pass the deadline without any call having
been cut. The LB cap break, which shares that closing line, closes the
same way.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

A walk the budget cut short closed on "see the section line above",
but that line goes only to capture-notes.txt, while the closing line
also reaches the job log, where nothing sits above it to see. It now
names the file.

The budget knob replacement and the run without a timeout binary had no
test of their own; both are now pinned.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

dp_run credited a cutoff to the budget whenever the deadline had passed
by the time the call returned. A call under its own, unshortened bound
can fire, or come back from its kill grace, just past a deadline it
never reached; the section note then said the budget cut a call, next
to the read's own note naming its own timeout, and the walk closed as
cut short by the budget. The bound alone is not enough either: a call
killed early, as an OOM-killed remote process is, returns 137 under a
bound cut to the deadline long before the deadline arrives.

dp_run and pod_on_node now share one rule: a cutoff is the budget's
only when the call's bound reached the deadline and the deadline has
passed, and never when no timeout binary bounded the call at all.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

An exec block keeps whatever it printed before its bound fired, and
nothing in the block said the bound had fired. The service map every
affected node gets is the largest such block, and a
truncated one answers the question it is there for, whether a backend
is still in the map, wrongly while looking complete.

dp_run now marks a cutoff where the call's output goes, as it already
does a refusal: the deadline's own cut names the budget, any other
names the call's bound. The section note also stops asserting that a
call was cut when the budget only refused the next one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The Ready-conditions and events reads share one block and added a line
of their own when they failed, on top of what their stderr put into
the block. Once dp_run marked every cut and refusal there, a cut read
carried two lines for one event, and before the budget status existed
the second named a timeout the first said the deadline had chosen.

The two reads now send stderr to the scratch file and, after their
output, write kubectl's stderr into the block whole, without dp_run's
marker, plus one line naming the read when it failed. kubectl's
message is kept as it came, since an error can follow lines of
throttling notices on a degraded apiserver. Without a scratch file
their stderr goes into the block directly. The probe comments also say
what the "spent" outcome now covers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

The test that a call cut by its own bound just past the deadline is not
blamed on the budget ran a 3s bound inside a 4s pod section. A
whole-second tick of date +%s between DP_T0 and the first call left the
bound reaching the deadline, the collector then blamed the budget, as
it should, and the test failed. The pod section is now 10s against the
same 3s bound, so a tick cannot reach the deadline.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>