Skip to content

test(e2e): cilium-dbg bpf lb list runs only when a LoadBalancer Service is broken, so a ClusterIP failure never gets the one table that would explain it #3689

Closed

Description

hack/e2e-capture-dataplane.sh does capture the eBPF service map, but the command lives inside capture_lb_datapath, and that section is entered only when the collector finds a Service of type=LoadBalancer carrying an ingress IP whose address fails a live reachability probe. The gate is bound to the kind of Service that is broken, while the table it guards describes every service translation on the node.

The consequence shows up whenever the failure is a ClusterIP one. In this in-tree install failure the collector ran correctly and logged no Service type=LoadBalancer with an ingress IP -- skipping LB-datapath capture, so the service map was never read, while the failure under investigation was a pod that could not reach 10.96.0.1:443. The question "which backend did the VIP resolve to, and was the unresponsive one still in the map" had no answer in the bundle, and could only be approached indirectly by reading connection-tracking flags for the client pod.

What the artifact did carry made the gap concrete rather than theoretical. In the cozyreport artifact of that run (retrievable until 2026-11-07), the EndpointSlice for default/kubernetes lists all three apiservers ready: true, while conntrack on the client's node shows completed flows to two of them and, to the third, TxFlagsSeen=0x02 with RxFlagsSeen=0x00 and LastRxReport=0 -- a SYN out and nothing ever back, on two separate flows. That reframes the symptom: not an unreachable ClusterIP, but a ready backend that swallows SYNs while its peers answer, which makes reachability a per-connection coin flip and makes any successful probe worthless as evidence.

Proposal: read the service map on every run rather than only on the LoadBalancer path, so that the ClusterIP case is covered by the instrument that already exists. The natural home is beside the existing per-node captures, which run whenever there is an affected pod at all. Worth deciding at the same time whether the connection-tracking read should be able to filter on a service VIP and not only on a pod address, since the reconstruction above had to work backwards from the pod side.

Not urgent, and not a defect in what the collector does -- everything it was asked to capture, it captured. The defect is that a working instrument is switched off by a condition unrelated to the question it answers.

Activity

  1. added
    area/ciliumIssues or PRs related to Cilium (agent, operator, Gateway API, network policy, LB-IPAM)
    on Aug 19, 2026
  2. added
    kind/featureCategorizes issue or PR as related to a new feature
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/ciliumIssues or PRs related to Cilium (agent, operator, Gateway API, network policy, LB-IPAM)area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/featureCategorizes issue or PR as related to a new featurepriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions