A pod can be refused an address that another allocator has already handed out, because two IPAMs share one pod CIDR. The pod then sits in ContainerCreating until the address is retried elsewhere, and any suite waiting on it fails on a timeout that names the workload rather than the network.
Observed
Run 31338952082, suite mongodb, failing assertion:
v1/Endpoints/tenant-test/mongodb-test-rs0
* (subsets[0].addresses[0].ip != null): Invalid value: false: Expected value: true
The endpoint is empty because the pod never got a sandbox. Seven FailedCreatePodSandBox events on the same pod, across six distinct pod UIDs, each ending the same way:
plugin type="cilium-cni" failed (add): unable to create endpoint:
[PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:10.244.1.12 is already in use"
Each refusal names the same address. Seven sandbox IDs and six pod UIDs appear across them, so recreation did not move the pod off it. Not every recreation was refused, and the last one did get a sandbox: the pod was in PodInitializing with no podIP recorded when the suite ran out of budget. Cilium never relents on the address; whether a pod escapes it depends on what hands out the next one.
Mechanism
Both allocators are configured over the same prefix, and neither knows about the other's reservations.
Corrected after digging into a run's artifacts; the original wording below was wrong about which addresses collide. Pod addresses come from kube-ovn alone: the CNI chain in packages/system/cilium/templates/cni.yaml is kube-ovn (with ipam.type: kube-ovn) then portmap then cilium-cni, and packages/system/cilium/values-kubeovn.yaml sets chainingMode: generic-veth, chainingTarget: kube-ovn. Cilium is a chained plugin there: it is handed an address and builds an endpoint for it.
What ipam.mode: "kubernetes" (packages/system/cilium/values.yaml:22) feeds is cilium's own infrastructure addresses, taken from that node's Node.spec.podCIDR: the cilium_host router IP and, because enable-envoy-config is on for Gateway API, the ingress IP. Meanwhile packages/system/kubeovn/values.yaml:7 sets POD_CIDR: "10.244.0.0/16" and kube-ovn hands pod addresses out flat from that /16, with Subnet/ovn-default excluding only the gateway. A per-node slice of the /16 is a subset of the flat range, so kube-ovn can hand a pod the address cilium already reserved for itself, and cilium then refuses to build the endpoint.
The run above is exactly that: 10.244.1.12 is srv1's CiliumNode.spec.ingress.ipv4 and is held by a live reserved:ingress endpoint, while ovn-controller logs Claiming ... 10.244.1.12 for the pod. Six such addresses exist in a three-node cluster, two per node.
Why it looks like an application problem
The suite reports an empty Endpoints object, which reads as "the workload did not become ready". Nothing in that assertion mentions the network, so the natural next step is to go and look at the workload. The CNI evidence is a FailedCreatePodSandBox event on the pod, several screens away from the failing assert.
A related quiet variant exists and is worse to diagnose: when the address handed out happens to be the node's own cilium_host router IP, cilium raises no objection at all. The endpoint is created, the pod runs, and only host-to-pod traffic breaks, so probes fail and the investigation goes into the application. Upstream tracks the loud variant as cilium/cilium#46799.
Scope
Pre-existing and independent of any branch: the run above is a PR whose diff does not touch networking. The exposure is largest on whichever node takes the earliest workloads, before kube-ovn has claimed much of the range, which is consistent with this appearing in occasional runs rather than every one.
A pod can be refused an address that another allocator has already handed out, because two IPAMs share one pod CIDR. The pod then sits in
ContainerCreatinguntil the address is retried elsewhere, and any suite waiting on it fails on a timeout that names the workload rather than the network.Observed
Run
31338952082, suitemongodb, failing assertion:The endpoint is empty because the pod never got a sandbox. Seven
FailedCreatePodSandBoxevents on the same pod, across six distinct pod UIDs, each ending the same way:Each refusal names the same address. Seven sandbox IDs and six pod UIDs appear across them, so recreation did not move the pod off it. Not every recreation was refused, and the last one did get a sandbox: the pod was in
PodInitializingwith nopodIPrecorded when the suite ran out of budget. Cilium never relents on the address; whether a pod escapes it depends on what hands out the next one.Mechanism
Both allocators are configured over the same prefix, and neither knows about the other's reservations.
Corrected after digging into a run's artifacts; the original wording below was wrong about which addresses collide. Pod addresses come from kube-ovn alone: the CNI chain in
packages/system/cilium/templates/cni.yamliskube-ovn(withipam.type: kube-ovn) thenportmapthencilium-cni, andpackages/system/cilium/values-kubeovn.yamlsetschainingMode: generic-veth,chainingTarget: kube-ovn. Cilium is a chained plugin there: it is handed an address and builds an endpoint for it.What
ipam.mode: "kubernetes"(packages/system/cilium/values.yaml:22) feeds is cilium's own infrastructure addresses, taken from that node'sNode.spec.podCIDR: thecilium_hostrouter IP and, becauseenable-envoy-configis on for Gateway API, the ingress IP. Meanwhilepackages/system/kubeovn/values.yaml:7setsPOD_CIDR: "10.244.0.0/16"and kube-ovn hands pod addresses out flat from that/16, withSubnet/ovn-defaultexcluding only the gateway. A per-node slice of the/16is a subset of the flat range, so kube-ovn can hand a pod the address cilium already reserved for itself, and cilium then refuses to build the endpoint.The run above is exactly that:
10.244.1.12is srv1'sCiliumNode.spec.ingress.ipv4and is held by a livereserved:ingressendpoint, whileovn-controllerlogsClaiming ... 10.244.1.12for the pod. Six such addresses exist in a three-node cluster, two per node.Why it looks like an application problem
The suite reports an empty
Endpointsobject, which reads as "the workload did not become ready". Nothing in that assertion mentions the network, so the natural next step is to go and look at the workload. The CNI evidence is aFailedCreatePodSandBoxevent on the pod, several screens away from the failing assert.A related quiet variant exists and is worse to diagnose: when the address handed out happens to be the node's own
cilium_hostrouter IP, cilium raises no objection at all. The endpoint is created, the pod runs, and only host-to-pod traffic breaks, so probes fail and the investigation goes into the application. Upstream tracks the loud variant as cilium/cilium#46799.Scope
Pre-existing and independent of any branch: the run above is a PR whose diff does not touch networking. The exposure is largest on whichever node takes the earliest workloads, before kube-ovn has claimed much of the range, which is consistent with this appearing in occasional runs rather than every one.