networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from

Two allocators draw from one address range on the kubeovn-cilium datapath and neither knows about the other. kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium allocates nothing for pods, it is a chained plugin, but it does take its own per-node infrastructure addresses out of Node.spec.podCIDR, which kube-controller-manager slices from the Kubernetes cluster CIDR. Set both to the same prefix, the way every example shows, and kube-ovn will sooner or later hand a pod an address cilium is already holding.

What collides

The CNI chain is kube-ovn, then portmap, then cilium-cni (packages/system/cilium/templates/cni.yaml), with cni.chainingMode: generic-veth and cni.chainingTarget: kube-ovn (packages/system/cilium/values-kubeovn.yaml:10-11). cilium-cni reads the address kube-ovn already put on the veth and registers an endpoint for it. It never allocates.

ipam.mode: "kubernetes" (packages/system/cilium/values.yaml:22-23) still hands the agent a host-scope allocator over Node.spec.podCIDR, and the agent takes two addresses per node out of it: the router IP on cilium_host, published as CiliumInternalIP, and the per-node Ingress IP in CiliumNode.spec.ingress.ipv4, allocated whenever enable-envoy-config is on. It is on, because Gateway API is enabled.

Upstream has a third address of this kind, the cilium-health endpoint IP. The chart renders enable-endpoint-health-checking: "false" for every chaining mode other than portmap and none (packages/system/cilium/charts/cilium/templates/cilium-configmap.yaml:999-1003), so it does not exist here.

Both remaining addresses come from ipam.PoolDefault() in daemon/infraendpoints/infra_ip_allocation.go at cilium v1.19.5, the same host-scope allocator built over the node's pod CIDR in pkg/ipam/ipam.go:121-135.

On the other side, packages/core/platform/values.yaml:63 sets networking.podCIDR: "10.244.0.0/16", packages/core/platform/templates/bundles/system.yaml:8 passes it to kube-ovn as POD_CIDR, and the resulting ovn-default Subnet allocates flat from that /16 across all nodes with nothing but the gateway in excludeIps.

Evidence

From a CI run where a pod sat in ContainerCreating until its suite gave up, refused on the same address every time:

node pod CIDR	`cilium_host`	`CiliumNode.spec.ingress.ipv4`
10.244.1.0/24	10.244.1.182	10.244.1.12
10.244.0.0/24	10.244.0.243	10.244.0.104
10.244.2.0/24	10.244.2.14	10.244.2.193

cilium-dbg endpoint list on the first node shows reserved:ingress 10.244.1.12 ready. ovn-controller on that same node logs Claiming <mac> 10.244.1.12 for the stuck pod, once per sandbox attempt. Both sides name the address, and they were collected independently.

The failure is loud when the collision is with the Ingress IP, because the agent refuses the sandbox with [PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:10.244.1.12 is already in use". That message is not an IPAM error. It comes from a duplicate-identifier lookup in pkg/endpoint/api/endpoint_api_manager.go:172 at v1.19.5, where the endpoint manager already holds an endpoint under the key ipv4:<addr>.

A collision with the router IP is silent. The endpoint gets created, the pod runs, and only host-to-pod traffic breaks, because to the node's own kernel that address is local.

Six addresses on a three-node cluster that a pod must never receive, at offsets nobody can predict: host-scope IPAM picks a random offset in the range and scans forward from there.

Not the upstream restart race

cilium/cilium#46799 describes a narrower version of this, where an agent restart rebuilds the host-scope bitmap empty and a pod ADD in that window can be handed the Ingress IP. No restart is needed here. kube-ovn has no notion of Node.spec.podCIDR at all, and I could not find a single read or write of it anywhere in kube-ovn v1.15.10 outside changelog text, so it can hand out a cilium infrastructure address at any moment.

Proposed fix

Move cilium's allocator off the shared range: ipam.mode: cluster-pool with ipam.operator.clusterPoolIPv4PodCIDRList pointing at a prefix that overlaps nothing.

kube-ovn prescribes the same thing for this exact chaining setup. makefiles/kind.mk:620-634 at kube-ovn v1.15.10 installs cilium with --set ipam.mode=cluster-pool, --set-json ipam.operator.clusterPoolIPv4PodCIDRList='["100.65.0.0/16"]', --set cni.chainingMode=generic-veth and --set cni.chainingTarget=kube-ovn, a pool deliberately unrelated to both kube-ovn's POD_CIDR and the Kubernetes cluster CIDR. One difference matters: upstream also passes --set envoy.enabled=false, so no Ingress IP is ever allocated there. Gateway API is not optional for us, so the Ingress IP is real and has to come from the separate pool too.

100.65.0.0/16 is free in this tree. 100.64.0.0/16 is the join CIDR and 100.66.0.0/16 is the transit CIDR.

Pod addresses are untouched by this. Cilium does not require a chained endpoint's address to sit inside its own allocation CIDR: in the run above, the node whose pod CIDR is 10.244.1.0/24 hosts pods on 10.244.0.65, 10.244.0.35 and 10.244.0.26. The allocation CIDR constrains the agent's own addresses and nothing else.

Under cluster-pool the agent reads CiliumNode.spec.ipam.podCIDRs instead of the Kubernetes Node (daemon/k8s/init.go:37-56 and pkg/node/types/node.go:76-81 at v1.19.5) and builds the same host-scope allocator over it, so the router and Ingress addresses follow the pool. No bootstrap deadlock: cilium-operator runs with hostNetwork: true, so it does not need the CNI it is configuring.

Why this is more than a values change

On an existing cluster the mode switch on its own does nothing, and it fails without saying so.

In kubernetes mode the agent writes CiliumNode.spec.ipam.podCIDRs itself, copying Node.spec.podCIDR (pkg/nodediscovery/nodediscovery.go:341-358 at v1.19.5). After the switch that field is already populated, and the operator will not overwrite it:

allocateNode sees a non-empty spec.ipam.podCIDRs and takes the reuse branch instead of allocating (pkg/ipam/allocator/podcidr/podcidr.go:466 and :489-499).
reuseIPNets hands the inherited CIDR to allocateIPNet (:677-681).
allocateIPNet rejects it, because the old /24 is not inside the new pool: if !cidrSet.InRange(newCIDR) { err = fmt.Errorf("allocator not configured for the requested CIDR %s", newCIDR); continue } (:802-806), and returns that error (:834-836).
The deferred block in allocateNode copies the error into Status.IPAM.OperatorStatus.Error, sets err = nil, and returns allocated = false (:442-464). The spec is left alone.

So the rendered values carry the new mode, the release is healthy, pods keep running, and the agent keeps allocating out of the old /24. The only trace is a string in CiliumNode.status.ipam.operator-status.error, and nobody has a reason to read it.

The comment above the write in nodediscovery.go promises the opposite, that "in other PodCIDR based IPAM modes (such as ClusterPool or MultiPool), the operator will set the PodCIDRs of the CiliumNode". It does, but only where the field is empty.

The change therefore needs a migration that clears spec.ipam.podCIDRs on every CiliumNode, and the ordering matters: it has to run after the agents are already on the new mode. Clear it any earlier and an agent still in kubernetes mode writes it straight back.

Cost of that: between the clear and the operator's allocation, an agent in cluster-pool mode has no allocation CIDR, so new pods on that node cannot get an endpoint. Running pods are unaffected.

This part is read from the sources, not tested. Someone should reproduce the switch on a live cluster before the migration is written.

Documentation is part of the fix

Nothing in docs/ says the Kubernetes cluster CIDR and networking.podCIDR must not overlap, and podSubnets is not mentioned at all. An operator working from the current material has no way to learn the constraint exists, and the obvious thing to do is exactly what triggers this. However the fix lands, the requirement has to be written down.

Alternatives considered

excludeIps on the ovn-default subnet. Cilium picks its offsets by random scan over the node's range, so there is no static set to exclude: the observed offsets were .182, .12, .243, .104, .14 and .193. Excluding a band wide enough to be safe is separating the ranges under another name. --default-exclude-ips is also only read when the subnet is created (pkg/controller/init.go:159 at kube-ovn v1.15.10); after that the Subnet has to be edited directly.
Changing POD_CIDR. Once ovn-default exists the configured value is ignored at startup for a same-family change, because initDefaultLogicalSwitch returns early and leaves spec.cidrBlock alone (pkg/controller/init.go:127-142 at v1.15.10). Editing the Subnet does work, but then every existing pod address falls outside the new block.
--ipv4-range. It does override the allocation CIDR, and it does beat Node.spec.podCIDR, because it is applied after node information is read (daemon/cmd/ipam_init.go:29-40 against daemon/cmd/daemon.go:158). But it is one value for every agent, so all nodes would allocate from the same range with no coordination between them, and the router IP is published cluster-wide. Making it per-node means generating a CiliumNodeConfig per node.
ipam.mode: delegated-plugin. Refused at startup together with Gateway API: checkIPAMDelegatedPlugin requires --enable-envoy-config to be off, and an explicit --local-router-ipv4 on top (pkg/option/config.go:3002-3009 at v1.19.5).
Changing the Kubernetes cluster CIDR. This works, and a separate e2e-only change is being prepared that does exactly that for the sandbox, but for a real installation that value lives in the operator's own machine configuration, outside this repository. It fixes CI and leaves the product as it is, which is why this issue exists separately from it.

The documentation gap in the section above understates this. The Talos bootstrap preset does not merely leave an operator uninformed, it hands them the overlapping configuration.

cozystack/talm, charts/cozystack/values.yaml:

podSubnets:
- 10.244.0.0/16
serviceSubnets:
- 10.96.0.0/16

and allocateNodeCIDRs: true further down the same file. That is the cluster CIDR kube-controller-manager slices Node.spec.podCIDR from, and it is byte-identical to the platform's own networking.podCIDR default. Anyone following the documented Talos path arrives at the collision, without deviating from the instructions anywhere.

The k3s path does not have this, and by coincidence rather than design: cozystack/ansible-cozystack, examples/ubuntu/inventory.yml sets cluster-cidr: 10.42.0.0/16 and service-cidr: 10.43.0.0/16, which are k3s defaults and happen to sit outside the platform range.

So the fix has a third part beyond the mode switch and the migration: the preset that ships the colliding value has to stop shipping it, or the platform default has to move out of its way. Whichever side moves, the two repositories have to agree, and nothing today makes them.

One more thing this changes, and it is worth recording separately: the mechanism written down in the tree for the Ingress-IP collision is not the one that happens here.

Two places say it. hack/e2e-cilium-endpoint-leak-healer.sh calls it a "host-scope IPAM restore-window race" in its header and spells it out at the gate that skips the agent restart, and hack/e2e-chainsaw/kuberture/probes.yaml:6-7 repeats it as the reason those probes are Jobs rather than bare Pods. Both describe a cilium-internal race: the agent restarts, the host-scope bitmap is rebuilt empty, the Ingress IP is re-reserved through a path that is not ordered against the CNI server, and a pod ADD served in that window is handed the Ingress IP.

That race is real in a cluster where cilium does its own pod IPAM. It is not what happens in this platform, because cilium does no pod IPAM here at all. It is chained behind kube-ovn (packages/system/cilium/templates/cni.yaml, cni.chainingMode: generic-veth with chainingTarget: kube-ovn in packages/system/cilium/values-kubeovn.yaml), so pod addresses come from kube-ovn and cilium only registers an endpoint for whatever address it is handed.

No restart window is needed for the collision. kube-ovn allocates flat from POD_CIDR and has no notion of Node.spec.podCIDR at all (I could not find a single read or write of it anywhere in kube-ovn v1.15.10 outside changelog text), so the address cilium reserved for its own Ingress endpoint is, from kube-ovn's side, simply the next free address in the subnet. It can be handed out at any moment, on a cluster that has never restarted an agent.

The evidence from the run this issue was filed against names both sides. cilium-dbg endpoint list on the affected node shows reserved:ingress 10.244.1.12 ready, matching that node's CiliumNode.spec.ingress.ipv4; ovn-controller on the same node logs Claiming <mac> 10.244.1.12 for the pod that never got a sandbox, once per attempt.

Why this is worth a note rather than a shrug: the wrong root is what made the hazard look unfixable. A cilium-internal race with no released fix leaves you writing a watchdog, an RBAC rule for it, a unit test for its decision helper, and reshaping a Chainsaw suite so its probes survive being deleted. A configuration overlap between two allocators is a config change. Four artifacts exist because of an attribution, not because of the defect.

The same misattribution reached upstream. The report I filed as cilium/cilium#46799 describes the restore-window race from source and cites this platform as the motivating downstream case. The source analysis in it does not depend on us and, as far as I can tell, still stands on its own for a cluster running cilium's own IPAM. The downstream motivation does not: what we hit was this overlap, not that window. That is worth correcting there so the issue is not carried by a reproduction that turns out to be a different bug.

Correction to my earlier comment: I got the k3s path wrong, and the two paths are not different.

I wrote that the k3s route escapes the overlap because its cluster CIDR is 10.42.0.0/16. I compared that against the platform chart's default networking.podCIDR of 10.244.0.0/16, which is the wrong pair. The Ansible role does not use the chart default: roles/cozystack/defaults/main.yml sets cozystack_pod_cidr: "10.42.0.0/16" and templates/platform-package.yml.j2 renders it straight into networking.podCIDR. The k3s inventories set cluster-cidr: 10.42.0.0/16. The two are identical, so that path carries the same collision, on a different prefix.

The reason both paths land there is written down. install/kubernetes/generic.md states as a warning that podCIDR and serviceCIDR must match your Kubernetes cluster configuration, and lists 10.42.0.0/16 for k3s and RKE2 and 10.244.0.0/16 for kubeadm. Equality is the documented requirement, and equality is the defect. An installation only escapes by not following that line.

The Talos pages prescribe the same shape from the other end. install/kubernetes/talosctl.md:123-124 and install/kubernetes/talos-bootstrap.md:99-100 set podSubnets: 10.244.0.0/16, and install/providers/hetzner.md sets the same podSubnets at 179-180 and then, at 356-357, sets podCIDR: "10.244.0.0/16" under the comment ## podSubnets from the node config. The released copies carry it too, not only trunk.

So the guidance has to change alongside whatever the platform does, and it is the part with the widest reach: it is what an operator follows on day one.

fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#3750

Four more instances, attributed with mechanisms while triaging the runs collected under #3570 (full evidence in #3570 (comment)). Silent router IP mode: runs 30971312858 (vmalertmanager-alertmanager-0 on the CiliumInternalIP of srv3), 31003636724 (cdi-operator on the one of srv1, where the collision also eats the apiserver reply to the pod and surfaces as a 30s discovery i/o timeout), 31003688705 (keycloak-db-2 on the one of srv3, startup probe refused 61 times while postgres inside streamed WAL). Loud ingress IP mode: run 31067501623 (vmselect-longterm-0 refused four sandboxes with putEndpointIdInvalid, the address is CiliumNode.spec.ingress.ipv4 of srv1). All four are from 2026-08-05 and 06. In the two runs from 2026-08-17 and 18 checked the same way, cilium infrastructure addresses sit in 100.65.0.0/16 and collide with none of the pod IPs, so newer snapshots look clean.

Dating note for the four instances above, so they are not read as the class burning in CI today. All four are from 2026-08-05 and 06, before the e2e sandbox moved its Talos pod subnet to 100.65.0.0/16 (with a guard keeping the sandbox CIDRs disjoint), which is why the two snapshots from 2026-08-17 and 18 show cilium infrastructure addresses outside the pod CIDR and no collisions. That separation covers the CI sandbox only: the shipped default is still podCIDR: "10.244.0.0/16" in the platform values while the cluster's own pod subnet is configured outside the chart, so nothing prevents the overlap for a user, which is exactly the opening statement of this issue. The defect stands; the CI reproducer no longer exists by construction.

Reproduced on a fresh install, not an upgrade: a 3-node arm64 Talos cluster (Talos v1.13.6, Kubernetes v1.36.5) on its first Cozystack install. The velero pod never got a sandbox: putEndpointIdInvalid "IP ipv4:10.244.0.81 is already in use", and cilium-dbg endpoint list on that node shows reserved:ingress 10.244.0.81 ready. So nothing about upgrade ordering is needed to hit it; one pod landing on the Ingress IP during the first install is enough.

hit this on real clusters too, not only in ci. putting what we see on record.

our setup is cozystack v1.6.4 on talos v1.13.6, three clusters on kubernetes v1.33 and v1.34, kubeovn-cilium variant with defaults. ipam=kubernetes, cni-chaining-mode=generic-veth, enable-envoy-config=true, ovn-default is 10.244.0.0/16 with only the gateway in excludeIps, and the node podCIDRs are /24s out of the same /16. kube-ovn has 186 to 212 addresses in use per cluster, so pods land on cilium addresses early.

four hits in the last four weeks, both modes from the issue.

silent router ip on its own node. a fresh postgres app never came up. the instance pod got 10.244.4.251, the CiliumInternalIP of the node it ran on. kubelet probes from that node hit cilium_host instead of the pod, Startup probe failed ... connect: connection refused 271 times, the cnpg cluster sat in Waiting for the instances to become active and the helm release timed out. postgres inside was up and accepting connections the whole time.
loud ingress ip on its own node. a helm post-delete hook pod (mariadb cleanup job) got 10.244.6.150, the CiliumNode.spec.ingress.ipv4 of its node. every sandbox attempt for 4 minutes was refused with putEndpointIdInvalid "IP ipv4:10.244.6.150 is already in use", the job hit its deadline, the uninstall ended with an error and the pvcs stayed behind.
router ip of another node. a virt-operator pod has held 10.244.0.69, the CiliumInternalIP of a different node, since 2026-09-10. it shows ready, but that node's host network can't reach it, and the kube-apiserver on that node calls the operator's validating webhook through this endpoint.
ingress ip of another node. a keycloak-operator pod on a second production cluster has held 10.244.0.206, the CiliumNode.spec.ingress.ipv4 of a different node, since 2026-10-02. also shows ready.

workaround for 1 was adding the six cilium addresses (router and ingress per node) to excludeIps on ovn-default and deleting the pod. it came back on a free address and the cluster went 2/2 ready in under a minute. that only covers the addresses cilium holds right now, a new node or a re-allocated address needs another edit, same point as alternative 1 above.

one more data point on the proposed fix. we tried cluster-pool with 100.65.0.0/16 on a 3 node test cluster. the migration itself went fine, but gateway api broke across nodes. envoy connects to backends from the node's ingress ip, and kube-ovn masquerades host originated traffic only when the source is in ovn40subnets (10.244.0.0/16 and 100.64.0.0/16 there). a 100.65.x source leaves unmasqueraded and the backend's reply goes to the backend's own node and dies. a nodeport through the gateway answered 200 only on the backend's node and 503 upstream connect error ... connection timeout on the other two. a pool cut from the end of the pod cidr and excluded from kube-ovn (10.244.248.0/21, /29 per node) keeps that source inside ovn40subnets and answered 200 from all three nodes. pr with that version is on the way.

fix(cilium)!: take cilium's own addresses from a block kube-ovn does …

[Backport release-1.6] fix(cilium)!: take cilium's own addresses from…

Navigation Menu

Repository navigation

networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740

What collides

Evidence

Not the upstream restart race

Proposed fix

Why this is more than a values change

Documentation is part of the fix

Alternatives considered

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740

Description

What collides

Evidence

Not the upstream restart race

Proposed fix

Why this is more than a values change

Documentation is part of the fix

Alternatives considered

Activity

lexfrei commented on Aug 10, 2026

lexfrei commented on Aug 10, 2026

lexfrei commented on Aug 10, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Sep 26, 2026

yankawai commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions