TLS passthrough through the tenant Gateway never connects when the backend is a hostNetwork pod on the same node as the cilium-envoy that handles the connection. The same backend on any other node connects every time.
I hit it on a three-node test cluster (isp-full, Kube-OVN chained with Cilium) and isolated it with a TLSRoute to a selectorless Service that has a single kube-apiserver endpoint:
| Client pod on |
apiserver on node A |
apiserver on node B |
apiserver on node C |
| node A |
0/10 |
10/10 |
10/10 |
| node C |
10/10 |
10/10 |
0/10 |
The client times out in the TLS handshake (curl exit 28). cilium-dbg monitor --type drop shows nothing. Cilium 1.19.5 and 1.20.2 (the #3343 and #4638 stack) behave the same, so this is not a 1.20 regression.
This hits the platform's own api.<host> route. It passes through to the kubernetes Service, and its endpoints are the control-plane apiservers on host network. In the test about 10 of 30 requests to api.<host> hung, which matches one of three endpoints being on the client's node. External clients probably see the same on the node that announces the Gateway address.
I don't have the mechanism yet. My guess is that Envoy's upstream connection to a local host address takes a path where the reply skips the proxy, but nothing proves it. The next step is a full cilium-dbg monitor trace and conntrack for one failing connection.
Related: #3913.
TLS passthrough through the tenant Gateway never connects when the backend is a hostNetwork pod on the same node as the cilium-envoy that handles the connection. The same backend on any other node connects every time.
I hit it on a three-node test cluster (isp-full, Kube-OVN chained with Cilium) and isolated it with a TLSRoute to a selectorless Service that has a single kube-apiserver endpoint:
The client times out in the TLS handshake (curl exit 28).
cilium-dbg monitor --type dropshows nothing. Cilium 1.19.5 and 1.20.2 (the #3343 and #4638 stack) behave the same, so this is not a 1.20 regression.This hits the platform's own
api.<host>route. It passes through to thekubernetesService, and its endpoints are the control-plane apiservers on host network. In the test about 10 of 30 requests toapi.<host>hung, which matches one of three endpoints being on the client's node. External clients probably see the same on the node that announces the Gateway address.I don't have the mechanism yet. My guess is that Envoy's upstream connection to a local host address takes a path where the reply skips the proxy, but nothing proves it. The next step is a full
cilium-dbg monitortrace and conntrack for one failing connection.Related: #3913.