Repository navigation
networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740
Description
Activity
The documentation gap in the section above understates this. The Talos bootstrap preset does not merely leave an operator uninformed, it hands them the overlapping configuration.
cozystack/talm,charts/cozystack/values.yaml:podSubnets: - 10.244.0.0/16 serviceSubnets: - 10.96.0.0/16
and
allocateNodeCIDRs: truefurther down the same file. That is the cluster CIDR kube-controller-manager slicesNode.spec.podCIDRfrom, and it is byte-identical to the platform's ownnetworking.podCIDRdefault. Anyone following the documented Talos path arrives at the collision, without deviating from the instructions anywhere.The k3s path does not have this, and by coincidence rather than design:
cozystack/ansible-cozystack,examples/ubuntu/inventory.ymlsetscluster-cidr: 10.42.0.0/16andservice-cidr: 10.43.0.0/16, which are k3s defaults and happen to sit outside the platform range.So the fix has a third part beyond the mode switch and the migration: the preset that ships the colliding value has to stop shipping it, or the platform default has to move out of its way. Whichever side moves, the two repositories have to agree, and nothing today makes them.
One more thing this changes, and it is worth recording separately: the mechanism written down in the tree for the Ingress-IP collision is not the one that happens here.
Two places say it.
hack/e2e-cilium-endpoint-leak-healer.shcalls it a "host-scope IPAM restore-window race" in its header and spells it out at the gate that skips the agent restart, andhack/e2e-chainsaw/kuberture/probes.yaml:6-7repeats it as the reason those probes are Jobs rather than bare Pods. Both describe a cilium-internal race: the agent restarts, the host-scope bitmap is rebuilt empty, the Ingress IP is re-reserved through a path that is not ordered against the CNI server, and a pod ADD served in that window is handed the Ingress IP.That race is real in a cluster where cilium does its own pod IPAM. It is not what happens in this platform, because cilium does no pod IPAM here at all. It is chained behind kube-ovn (
packages/system/cilium/templates/cni.yaml,cni.chainingMode: generic-vethwithchainingTarget: kube-ovninpackages/system/cilium/values-kubeovn.yaml), so pod addresses come from kube-ovn and cilium only registers an endpoint for whatever address it is handed.No restart window is needed for the collision. kube-ovn allocates flat from
POD_CIDRand has no notion ofNode.spec.podCIDRat all (I could not find a single read or write of it anywhere in kube-ovn v1.15.10 outside changelog text), so the address cilium reserved for its own Ingress endpoint is, from kube-ovn's side, simply the next free address in the subnet. It can be handed out at any moment, on a cluster that has never restarted an agent.The evidence from the run this issue was filed against names both sides.
cilium-dbg endpoint liston the affected node showsreserved:ingress 10.244.1.12 ready, matching that node'sCiliumNode.spec.ingress.ipv4;ovn-controlleron the same node logsClaiming <mac> 10.244.1.12for the pod that never got a sandbox, once per attempt.Why this is worth a note rather than a shrug: the wrong root is what made the hazard look unfixable. A cilium-internal race with no released fix leaves you writing a watchdog, an RBAC rule for it, a unit test for its decision helper, and reshaping a Chainsaw suite so its probes survive being deleted. A configuration overlap between two allocators is a config change. Four artifacts exist because of an attribution, not because of the defect.
The same misattribution reached upstream. The report I filed as cilium/cilium#46799 describes the restore-window race from source and cites this platform as the motivating downstream case. The source analysis in it does not depend on us and, as far as I can tell, still stands on its own for a cluster running cilium's own IPAM. The downstream motivation does not: what we hit was this overlap, not that window. That is worth correcting there so the issue is not carried by a reproduction that turns out to be a different bug.
Correction to my earlier comment: I got the k3s path wrong, and the two paths are not different.
I wrote that the k3s route escapes the overlap because its cluster CIDR is
10.42.0.0/16. I compared that against the platform chart's defaultnetworking.podCIDRof10.244.0.0/16, which is the wrong pair. The Ansible role does not use the chart default:roles/cozystack/defaults/main.ymlsetscozystack_pod_cidr: "10.42.0.0/16"andtemplates/platform-package.yml.j2renders it straight intonetworking.podCIDR. The k3s inventories setcluster-cidr: 10.42.0.0/16. The two are identical, so that path carries the same collision, on a different prefix.The reason both paths land there is written down.
install/kubernetes/generic.mdstates as a warning thatpodCIDRandserviceCIDRmust match your Kubernetes cluster configuration, and lists10.42.0.0/16for k3s and RKE2 and10.244.0.0/16for kubeadm. Equality is the documented requirement, and equality is the defect. An installation only escapes by not following that line.The Talos pages prescribe the same shape from the other end.
install/kubernetes/talosctl.md:123-124andinstall/kubernetes/talos-bootstrap.md:99-100setpodSubnets: 10.244.0.0/16, andinstall/providers/hetzner.mdsets the samepodSubnetsat 179-180 and then, at 356-357, setspodCIDR: "10.244.0.0/16"under the comment## podSubnets from the node config. The released copies carry it too, not only trunk.So the guidance has to change alongside whatever the platform does, and it is the part with the widest reach: it is what an operator follows on day one.
- added a commit that references this issue
on Aug 11, 2026 Four more instances, attributed with mechanisms while triaging the runs collected under #3570 (full evidence in #3570 (comment)). Silent router IP mode: runs 30971312858 (vmalertmanager-alertmanager-0 on the CiliumInternalIP of srv3), 31003636724 (cdi-operator on the one of srv1, where the collision also eats the apiserver reply to the pod and surfaces as a 30s discovery i/o timeout), 31003688705 (keycloak-db-2 on the one of srv3, startup probe refused 61 times while postgres inside streamed WAL). Loud ingress IP mode: run 31067501623 (vmselect-longterm-0 refused four sandboxes with putEndpointIdInvalid, the address is CiliumNode.spec.ingress.ipv4 of srv1). All four are from 2026-08-05 and 06. In the two runs from 2026-08-17 and 18 checked the same way, cilium infrastructure addresses sit in 100.65.0.0/16 and collide with none of the pod IPs, so newer snapshots look clean.
Dating note for the four instances above, so they are not read as the class burning in CI today. All four are from 2026-08-05 and 06, before the e2e sandbox moved its Talos pod subnet to 100.65.0.0/16 (with a guard keeping the sandbox CIDRs disjoint), which is why the two snapshots from 2026-08-17 and 18 show cilium infrastructure addresses outside the pod CIDR and no collisions. That separation covers the CI sandbox only: the shipped default is still
podCIDR: "10.244.0.0/16"in the platform values while the cluster's own pod subnet is configured outside the chart, so nothing prevents the overlap for a user, which is exactly the opening statement of this issue. The defect stands; the CI reproducer no longer exists by construction.Reproduced on a fresh install, not an upgrade: a 3-node arm64 Talos cluster (Talos v1.13.6, Kubernetes v1.36.5) on its first Cozystack install. The velero pod never got a sandbox:
putEndpointIdInvalid "IP ipv4:10.244.0.81 is already in use", andcilium-dbg endpoint liston that node showsreserved:ingress 10.244.0.81 ready. So nothing about upgrade ordering is needed to hit it; one pod landing on the Ingress IP during the first install is enough.hit this on real clusters too, not only in ci. putting what we see on record.
our setup is cozystack v1.6.4 on talos v1.13.6, three clusters on kubernetes v1.33 and v1.34, kubeovn-cilium variant with defaults.
ipam=kubernetes,cni-chaining-mode=generic-veth,enable-envoy-config=true,ovn-defaultis10.244.0.0/16with only the gateway inexcludeIps, and the nodepodCIDRs are /24s out of the same /16. kube-ovn has 186 to 212 addresses in use per cluster, so pods land on cilium addresses early.four hits in the last four weeks, both modes from the issue.
- silent router ip on its own node. a fresh postgres app never came up. the instance pod got
10.244.4.251, theCiliumInternalIPof the node it ran on. kubelet probes from that node hitcilium_hostinstead of the pod,Startup probe failed ... connect: connection refused271 times, the cnpg cluster sat inWaiting for the instances to become activeand the helm release timed out. postgres inside was up and accepting connections the whole time. - loud ingress ip on its own node. a helm post-delete hook pod (mariadb cleanup job) got
10.244.6.150, theCiliumNode.spec.ingress.ipv4of its node. every sandbox attempt for 4 minutes was refused withputEndpointIdInvalid "IP ipv4:10.244.6.150 is already in use", the job hit its deadline, the uninstall ended with an error and the pvcs stayed behind. - router ip of another node. a
virt-operatorpod has held10.244.0.69, theCiliumInternalIPof a different node, since 2026-09-10. it shows ready, but that node's host network can't reach it, and the kube-apiserver on that node calls the operator's validating webhook through this endpoint. - ingress ip of another node. a
keycloak-operatorpod on a second production cluster has held10.244.0.206, theCiliumNode.spec.ingress.ipv4of a different node, since 2026-10-02. also shows ready.
workaround for 1 was adding the six cilium addresses (router and ingress per node) to
excludeIpsonovn-defaultand deleting the pod. it came back on a free address and the cluster went 2/2 ready in under a minute. that only covers the addresses cilium holds right now, a new node or a re-allocated address needs another edit, same point as alternative 1 above.one more data point on the proposed fix. we tried
cluster-poolwith100.65.0.0/16on a 3 node test cluster. the migration itself went fine, but gateway api broke across nodes. envoy connects to backends from the node's ingress ip, and kube-ovn masquerades host originated traffic only when the source is inovn40subnets(10.244.0.0/16and100.64.0.0/16there). a100.65.xsource leaves unmasqueraded and the backend's reply goes to the backend's own node and dies. a nodeport through the gateway answered 200 only on the backend's node and 503upstream connect error ... connection timeouton the other two. a pool cut from the end of the pod cidr and excluded from kube-ovn (10.244.248.0/21, /29 per node) keeps that source insideovn40subnetsand answered 200 from all three nodes. pr with that version is on the way.- silent router ip on its own node. a fresh postgres app never came up. the instance pod got
- added a commit that references this issue
on Oct 8, 2026
Two allocators draw from one address range on the kubeovn-cilium datapath and neither knows about the other. kube-ovn owns pod IPAM and allocates flat from
POD_CIDR. Cilium allocates nothing for pods, it is a chained plugin, but it does take its own per-node infrastructure addresses out ofNode.spec.podCIDR, which kube-controller-manager slices from the Kubernetes cluster CIDR. Set both to the same prefix, the way every example shows, and kube-ovn will sooner or later hand a pod an address cilium is already holding.What collides
The CNI chain is
kube-ovn, thenportmap, thencilium-cni(packages/system/cilium/templates/cni.yaml), withcni.chainingMode: generic-vethandcni.chainingTarget: kube-ovn(packages/system/cilium/values-kubeovn.yaml:10-11). cilium-cni reads the address kube-ovn already put on the veth and registers an endpoint for it. It never allocates.ipam.mode: "kubernetes"(packages/system/cilium/values.yaml:22-23) still hands the agent a host-scope allocator overNode.spec.podCIDR, and the agent takes two addresses per node out of it: the router IP oncilium_host, published asCiliumInternalIP, and the per-node Ingress IP inCiliumNode.spec.ingress.ipv4, allocated wheneverenable-envoy-configis on. It is on, because Gateway API is enabled.Upstream has a third address of this kind, the cilium-health endpoint IP. The chart renders
enable-endpoint-health-checking: "false"for every chaining mode other thanportmapandnone(packages/system/cilium/charts/cilium/templates/cilium-configmap.yaml:999-1003), so it does not exist here.Both remaining addresses come from
ipam.PoolDefault()indaemon/infraendpoints/infra_ip_allocation.goat cilium v1.19.5, the same host-scope allocator built over the node's pod CIDR inpkg/ipam/ipam.go:121-135.On the other side,
packages/core/platform/values.yaml:63setsnetworking.podCIDR: "10.244.0.0/16",packages/core/platform/templates/bundles/system.yaml:8passes it to kube-ovn asPOD_CIDR, and the resultingovn-defaultSubnet allocates flat from that/16across all nodes with nothing but the gateway inexcludeIps.Evidence
From a CI run where a pod sat in
ContainerCreatinguntil its suite gave up, refused on the same address every time:cilium_hostCiliumNode.spec.ingress.ipv4cilium-dbg endpoint liston the first node showsreserved:ingress 10.244.1.12 ready.ovn-controlleron that same node logsClaiming <mac> 10.244.1.12for the stuck pod, once per sandbox attempt. Both sides name the address, and they were collected independently.The failure is loud when the collision is with the Ingress IP, because the agent refuses the sandbox with
[PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:10.244.1.12 is already in use". That message is not an IPAM error. It comes from a duplicate-identifier lookup inpkg/endpoint/api/endpoint_api_manager.go:172at v1.19.5, where the endpoint manager already holds an endpoint under the keyipv4:<addr>.A collision with the router IP is silent. The endpoint gets created, the pod runs, and only host-to-pod traffic breaks, because to the node's own kernel that address is local.
Six addresses on a three-node cluster that a pod must never receive, at offsets nobody can predict: host-scope IPAM picks a random offset in the range and scans forward from there.
Not the upstream restart race
cilium/cilium#46799describes a narrower version of this, where an agent restart rebuilds the host-scope bitmap empty and a pod ADD in that window can be handed the Ingress IP. No restart is needed here. kube-ovn has no notion ofNode.spec.podCIDRat all, and I could not find a single read or write of it anywhere in kube-ovn v1.15.10 outside changelog text, so it can hand out a cilium infrastructure address at any moment.Proposed fix
Move cilium's allocator off the shared range:
ipam.mode: cluster-poolwithipam.operator.clusterPoolIPv4PodCIDRListpointing at a prefix that overlaps nothing.kube-ovn prescribes the same thing for this exact chaining setup.
makefiles/kind.mk:620-634at kube-ovn v1.15.10 installs cilium with--set ipam.mode=cluster-pool,--set-json ipam.operator.clusterPoolIPv4PodCIDRList='["100.65.0.0/16"]',--set cni.chainingMode=generic-vethand--set cni.chainingTarget=kube-ovn, a pool deliberately unrelated to both kube-ovn'sPOD_CIDRand the Kubernetes cluster CIDR. One difference matters: upstream also passes--set envoy.enabled=false, so no Ingress IP is ever allocated there. Gateway API is not optional for us, so the Ingress IP is real and has to come from the separate pool too.100.65.0.0/16is free in this tree.100.64.0.0/16is the join CIDR and100.66.0.0/16is the transit CIDR.Pod addresses are untouched by this. Cilium does not require a chained endpoint's address to sit inside its own allocation CIDR: in the run above, the node whose pod CIDR is
10.244.1.0/24hosts pods on10.244.0.65,10.244.0.35and10.244.0.26. The allocation CIDR constrains the agent's own addresses and nothing else.Under cluster-pool the agent reads
CiliumNode.spec.ipam.podCIDRsinstead of the Kubernetes Node (daemon/k8s/init.go:37-56andpkg/node/types/node.go:76-81at v1.19.5) and builds the same host-scope allocator over it, so the router and Ingress addresses follow the pool. No bootstrap deadlock: cilium-operator runs withhostNetwork: true, so it does not need the CNI it is configuring.Why this is more than a values change
On an existing cluster the mode switch on its own does nothing, and it fails without saying so.
In
kubernetesmode the agent writesCiliumNode.spec.ipam.podCIDRsitself, copyingNode.spec.podCIDR(pkg/nodediscovery/nodediscovery.go:341-358at v1.19.5). After the switch that field is already populated, and the operator will not overwrite it:allocateNodesees a non-emptyspec.ipam.podCIDRsand takes the reuse branch instead of allocating (pkg/ipam/allocator/podcidr/podcidr.go:466and:489-499).reuseIPNetshands the inherited CIDR toallocateIPNet(:677-681).allocateIPNetrejects it, because the old/24is not inside the new pool:if !cidrSet.InRange(newCIDR) { err = fmt.Errorf("allocator not configured for the requested CIDR %s", newCIDR); continue }(:802-806), and returns that error (:834-836).allocateNodecopies the error intoStatus.IPAM.OperatorStatus.Error, setserr = nil, and returnsallocated = false(:442-464). The spec is left alone.So the rendered values carry the new mode, the release is healthy, pods keep running, and the agent keeps allocating out of the old
/24. The only trace is a string inCiliumNode.status.ipam.operator-status.error, and nobody has a reason to read it.The comment above the write in
nodediscovery.gopromises the opposite, that "in other PodCIDR based IPAM modes (such as ClusterPool or MultiPool), the operator will set the PodCIDRs of the CiliumNode". It does, but only where the field is empty.The change therefore needs a migration that clears
spec.ipam.podCIDRson everyCiliumNode, and the ordering matters: it has to run after the agents are already on the new mode. Clear it any earlier and an agent still inkubernetesmode writes it straight back.Cost of that: between the clear and the operator's allocation, an agent in cluster-pool mode has no allocation CIDR, so new pods on that node cannot get an endpoint. Running pods are unaffected.
This part is read from the sources, not tested. Someone should reproduce the switch on a live cluster before the migration is written.
Documentation is part of the fix
Nothing in
docs/says the Kubernetes cluster CIDR andnetworking.podCIDRmust not overlap, andpodSubnetsis not mentioned at all. An operator working from the current material has no way to learn the constraint exists, and the obvious thing to do is exactly what triggers this. However the fix lands, the requirement has to be written down.Alternatives considered
excludeIpson theovn-defaultsubnet. Cilium picks its offsets by random scan over the node's range, so there is no static set to exclude: the observed offsets were.182,.12,.243,.104,.14and.193. Excluding a band wide enough to be safe is separating the ranges under another name.--default-exclude-ipsis also only read when the subnet is created (pkg/controller/init.go:159at kube-ovn v1.15.10); after that the Subnet has to be edited directly.POD_CIDR. Onceovn-defaultexists the configured value is ignored at startup for a same-family change, becauseinitDefaultLogicalSwitchreturns early and leavesspec.cidrBlockalone (pkg/controller/init.go:127-142at v1.15.10). Editing the Subnet does work, but then every existing pod address falls outside the new block.--ipv4-range. It does override the allocation CIDR, and it does beatNode.spec.podCIDR, because it is applied after node information is read (daemon/cmd/ipam_init.go:29-40againstdaemon/cmd/daemon.go:158). But it is one value for every agent, so all nodes would allocate from the same range with no coordination between them, and the router IP is published cluster-wide. Making it per-node means generating aCiliumNodeConfigper node.ipam.mode: delegated-plugin. Refused at startup together with Gateway API:checkIPAMDelegatedPluginrequires--enable-envoy-configto be off, and an explicit--local-router-ipv4on top (pkg/option/config.go:3002-3009at v1.19.5).