Repository navigation
e2e: pod IP becomes a local address of the node host netns, kubelet probes its own loopback #3615
Description
Summary
In four red e2e runs across three sandbox nodes and four unrelated components, the failing pod's IP turns out to be a local address of the node's host netns: ip route get <podIP> executed in the host netns answers local <podIP> dev lo table local src <podIP>. Kubelet runs its HTTP probes from the host netns, so every probe connects to the node's own loopback, where nothing listens on the probe port, and gets ECONNREFUSED forever. Depending on whether the pod has a liveness probe, the outcome is either a pod stuck 0/1 Ready or kubelet killing a perfectly healthy process in a CrashLoopBackOff, and either way the Deployment reports Failed, helm-controller gives up on stalled resources, and the install phase collapses behind it.
Fingerprint
| run | pod | node | ip route get <podIP> in host netns |
|---|---|---|---|
| nightly 2026-08-07 (31143191747) | mariadb-operator-webhook, 10.244.0.36 | srv2 | local 10.244.0.36 dev lo table local |
| PR run (31174949791) | snapshot-controller, 10.244.0.38 | srv1 | local 10.244.0.38 dev lo table local |
| PR run (31183108601) | kube-ovn-pinger, 10.244.0.7 | srv1 | local 10.244.0.7 dev lo table local |
| nightly 2026-08-05 (30971312858) | vmalertmanager-alertmanager-0, 10.244.0.140 | srv3 | local 10.244.0.140 dev lo table local |
Control from the same collector, a healthy pod on the same node in the same window: cert-controller at 10.244.0.37 answers 10.244.0.37 via 100.64.0.1 dev ovn0 src 192.168.123.11 and its conntrack is full of TIME_WAIT entries on the probe port.
The component is innocent, proven twice
In the first two cases the process inside the pod is demonstrably alive while its probes fail. The mariadb-operator webhook loaded its serving certs and reported Serving webhook server port 9443, then sat for 30 minutes with zero restarts and x360 Unhealthy ... connect: connection refused events, and Cilium's conntrack on the node holds no entries at all for its IP, so probe traffic never reached the pod's datapath. The snapshot-controller pod held live outbound TCP to the apiserver on 6443 while not a single inbound entry to its probe port 8080 existed; kubelet killed it nine times by liveness with a clean two-line log. A probe eaten by the host's loopback is the only reading consistent with both.
Why the local route wins
ip rule on all nodes is 100: from all lookup local, 32766: from all lookup main. The kube-ovn route to the pod lives in main, so once an address lands in the local table the host answers for it and the ovn0 path never gets consulted.
Causal boundary
The route anomaly is present in all four runs but causally proven only in the first two, where the process was alive. In the other two the pod exits on its own before binding its port (kube-ovn-pinger times out reaching the ClusterIP apiserver at 10.96.0.1 and exits 1 after its own 15s deadline), so the anomaly is a correlate there, not an established cause.
The collector's own leading hypothesis does not hold for these runs
hack/e2e-capture-dataplane.sh describes this exact symptom and attributes it to ovn-controller incremental-processing lag under CPU throttling. In both runs where its design evidence was checked: nr_throttled 0, throttled_usec 0, no physical_flow_output ... took Nms lines, no Unreasonably long ... poll interval lines, and the condition did not self-heal in 31, 14 or 16 minutes while the header describes a transient of several minutes. The throttling story does not fit; the question is L3: who installs a local address or local route for a pod IP.
Candidate root, not yet proven
Two IPAMs carve the same 10.244.0.0/16: kube-ovn assigns pod IPs from it flat, and Cilium holds its own reserved addresses from the same range (reserved:ingress was 10.244.0.66 on srv2 and 10.244.2.92 on srv1 in these runs). Two allocators over one CIDR can hand out the same address twice, and a Cilium-owned address is exactly the kind that lands in the host's local table. What is missing to name the owner: the collector captures only ip addr show ovn0, so the interface holding the stolen address is never recorded.
Instrumentation to settle it on the next occurrence
Two additions to hack/e2e-capture-dataplane.sh close the attribution gap: capture ip -o addr show for the whole host netns and ip route show table local. A third makes the evidence self-contained: also snapshot one known-healthy pod on the same node as an in-run control, since the collector currently captures NotReady pods only and the healthy-path baseline has to be borrowed from another run.
Activity
test(e2e): capture host-netns address attribution and a healthy-pod r…
lexfrei commentedon Aug 10, 2026
The interface is
cilium_host. Two runs from 2026-08-09 carry the capture #3617 added, and both name it.31338950312 (job 93311099197),
harbor-test-nginx-c69459d47-rnkjson srv3:31338954173 (job 93313889744),
harbor-test-db-1on srv2:So the colliding address is the node's own CiliumInternalIP, not the
reserved:ingressone this issue used as its example. Same origin, so they behave the same once handed out.The arithmetic closes in both runs. kube-controller-manager carves a /24 per node out of
10.244.0.0/16and cilium takes its router address from that /24: srv3 has10.244.0.0/24withcilium_hostat.197, srv2 has10.244.1.0/24withcilium_hostat.94.ovn-defaultiscidrBlock: 10.244.0.0/16withexcludeIps: ['10.244.0.1'], so kube-ovn allocates flat across the whole range and knows nothing about the per-node router IPs.One thing to settle before anyone tries to fix this with
excludeIps. The router address is not at a fixed offset:.197,.94and.104across the three nodes I have.pkg/ipam/hostscope.goallocates throughipallocator.NewCIDRRange, which builds anallocator.NewAllocationMap, and that is documented as random:randomScanStrategy.AllocateBitstarts atrand.IntN(max)and scans forward, so the address is spread evenly over the node's /24 and moves whenever the agent loses its state. Checked against cilium v1.19.5. A narrow static exclusion cannot cover it, and one wide enough to cover it is range separation wearing a smaller diff.One thing to add to the symptom description. The second run separates this from a workload that never opened its port. kubelet on srv2 gets
connection refusedonhttps://10.244.1.94:8000/startupzwhile cilium conntrack on the same node has established conversations with that address:Pod to pod reaches the port that host to pod is refused on. A dead process refuses both, so that pair rules the application out without reading its logs.
The loud variant hit the same night: mongodb in run 31338952082,
putEndpointIdInvalid "IP ipv4:10.244.1.12 is already in use"underFailedCreatePodSandBox. Three of the four suite failures in that batch of eight runs were this bug in one form or the other.#3736 has the root cause and the config behind it. The two issues are not linked right now.
fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#3750
lexfrei commentedon Sep 23, 2026
Same root cause as #3736: the
cilium_hostrouter IP came from the pod range kube-ovn allocates from. #3750 moved the sandbox cluster CIDR to 100.65.0.0/16 and addedhack/sandbox-cidr-disjoint.bats, so this can't happen in e2e anymore. The same overlap on real clusters is tracked in #3740.