fix(e2e): capture both ends of the worker byte path on node-join failure - #3804

What this PR does

A tenant worker that misses the node-Ready deadline while its kubelet image pull crawls leaves two mechanisms behind one symptom, and the failure artifact could not tell them apart: either the link is slow, or the pull keeps restarting and discarding what it fetched. From inside the guest both are a progress line that barely moves. This adds the host-side counters that separate them, and the guest-side reads that explain a kubelet which never got as far as pulling at all.

Host side. A per-worker network capture reads the eight container_network_* families from the same kubelet cAdvisor endpoint the CPU throttling capture already uses. cAdvisor builds them from the interface counters inside the Pod's own network namespace and publishes one row per interface, so what arrived at the worker is counted there in full, including the bytes a restart threw away, set against the progress the guest reported, that separates a pull which kept restarting from a slow link.

Which row answers is not obvious, and getting it wrong inverts the reading, so the collector states it rather than leaving it to the reader. A bridge-bound worker presents four interfaces in its Pod netns: the one still named eth0 is a dummy that holds the Pod IP so the kubelet's check passes and carries no traffic, so it reports zero, and it is the row a reader reaches for first, by name, where zero reads as a worker that received nothing. The number that answers is the renamed NIC's receive; the bridge counts the same bytes again, and the tap's counters are named from the host's end. The filter deliberately carries no interface predicate, so all four reach the artifact and can be told apart.

Two further properties of the family fail by capturing nothing rather than loudly, so both are stated where the filter is: the dropped-packet series are spelled packets_dropped and there is no *_drops_total, and every row carries an empty container label because cAdvisor publishes them on the Pod sandbox, which means a filter borrowed from the CPU counters beside it matches nothing at all.

The fixture's row shape and values are real kubelet output with the identifying labels rewritten. Its four interface names are derived rather than observed: no virt-launcher Pod was available to sample, so they are read from the name generators in the KubeVirt version this repo ships, and the counters attached to them illustrate direction rather than magnitude.

Guest side. The Talos capture gains the service state machine and the link table. Talos creates a service's log buffer when its runner opens the log writer, as it starts the process, so a kubelet still waiting on its volumes, on time sync, or on the container runtime reporting healthy never reached its runner and has no log at all, and talosctl logs kubelet then fails naming a log that was never registered, which is true about the log and silent about the service. talosctl services answers with the state and the condition being waited on, in one call for every service, so the runtime a kubelet is blocked behind arrives in the same file. The link table is read as YAML because MTU lives in the LinkStatus spec and not among the resource's printed columns, and it is the value that decides whether a path-MTU reading of a stalled transfer survives.

Both new reads are made with the short-lived os:reader credential the capture already mints; neither needs a wider role.

A worker that answered nothing now says so once, for the worker. Each read already carried its own error and exit code, which is accurate per file and adds up to a directory indistinguishable from a worker that simply had little to report.

What this does not do

This collects evidence; it fixes nothing and is not expected to change any suite's outcome. The comparison it enables is not symmetric either, and the source says so: the host side reports bytes on an interface, while the guest side reports an image pull's own progress, which is unpacking rather than arrival. They bound each other; they are not two measurements of one quantity.

Cost

The diagnostics phase budget drops from 8m to 7m. That is forced rather than chosen: the inequality that keeps the tenant crust-gather snapshot reachable had ten seconds of slack, and the two added guest reads cost sixty across the minimum worker pool. A shorter budget declines more, and what it declines first is what is gated last: the mirror capture, whose subject is partly recoverable elsewhere, and the image-cache re-probe, which has no substitute and was already first to go. The new network capture also spends a listing and one read per node ahead of the guest captures, so those reach their gate with less headroom than before. All of it is priced where the number lives, as a ceiling the read bounds impose rather than a duration measured on a live cluster, and a collector the budget declines says so in the log rather than going quiet.

The remaining slack is ten seconds, which is worth knowing before the next collector is added rather than after: it buys no bounded read at any bound this file uses, so adding one means re-deriving the budget again.

Testing

hack/run-kubernetes-network-counters_test.bats is new (15 tests). The node-join, Talos-diagnostics and CPU-throttle suites gain coverage for the phase ordering, the per-worker marker boundary, the spend-order comments in both Chainsaw suites, and the enumeration of collectors that survive a missing timeout. Every claim the two Chainsaw comment blocks make about spend order is derived from the source rather than restated, so a collector added without a line there fails.

The two cAdvisor captures now run one shared body. They previously carried near-identical copies, and the copies cost more than duplication normally does: the block-level bound audit pins each collector by something only it produces, and two collectors issuing identical reads left neither with anything of its own, so removing one from the audit went unnoticed. Each is now pinned by the report directory it writes.

Verified with hack/cozytest.sh across eleven suites, checking that the number of tests executed matches the number declared, because an exit code answers "did anything fail", never "did anything run". Each behavioural change was confirmed by mutating the source and observing the intended test fail.

Screenshots

No UI changes.

Downstream repositories

Walked the trigger map file by file against the diff. All seven changed files live under hack/, and the change adds one file and modifies six; no repository file is moved, renamed or deleted. The ccp entry is the only one that keys on hack/ at all, and it keys on hack/package.mk, hack/common-envs.mk and hack/update-crd.sh as anchors for make generate; none of those is touched, and no make target changes behaviour. Nothing here is a package, a chart value, an ApplicationDefinition, a CRD, a release asset, a namespace, a variant, or a node prerequisite, so no other entry applies.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

NONE

Summary by CodeRabbit

• New Features
  • Added worker network-counter diagnostics with interface-level and cumulative traffic data.
  • Expanded Talos diagnostics with service-state and link-state information.
• Bug Fixes
  • Improved reporting for partial captures, timeouts, unavailable data, and zero-traffic workers.
  • Clarified failure reporting and fallback messages when timeout controls are unavailable.
• Documentation
  • Updated diagnostic timing and collector documentation for the seven-minute budget and new metrics.
• Tests
  • Added comprehensive coverage for network counters, filtering, truncation, and expanded Talos diagnostic scenarios.