fix(e2e): stop the sandbox cluster CIDR overlapping the pod range - #3750

What this PR does

The sandbox Talos config and the platform install step both used 10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the kube-ovn POD_CIDR, so two allocators drew from one range.

kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is chained behind it (generic-veth, target kube-ovn) and allocates no pod addresses at all, but it still carves its own per-node router IP and Ingress IP out of Node.spec.podCIDR, which kube-controller-manager slices from the cluster CIDR. Sooner or later kube-ovn hands a pod an address cilium already holds. When that address is the Ingress IP the agent refuses the sandbox with IP ipv4:<addr> is already in use, and cilium never relents on it, so the pod proceeds only if something reschedules it onto a different address; in the run this was filed from, the same address was refused seven times across six pod UIDs before the suite ran out of budget. When it is the router IP nothing complains and only host-to-pod traffic breaks.

This moves the sandbox cluster CIDR to 100.65.0.0/16, which no other range in the tree uses, and adds a guard so the two cannot be brought back together by accident.

Pod addresses do not change. They still come from kube-ovn out of 10.244.0.0/16, and cilium does not require a chained endpoint's address to sit inside its own allocation CIDR, so only cilium's two per-node infrastructure addresses move.

The quiet variant gets fixed twice over, which is worth spelling out because only half of it was obvious. Under ipam.mode: kubernetes cilium also installs a route for the node's own pod CIDR via cilium_host. On the current default that route covers 10.244.a.0/24 on a node whose pods actually live behind ovn0, so it shadows real pod addresses. Moving the cluster CIDR both stops the router IP from ever being handed to a pod and retargets that route at 100.65.a.0/24, where it shadows nothing.

Excluding the addresses on the kube-ovn side instead is not an option: cilium picks its offsets by random scan over the node's range, so there is no static set to exclude.

The collision shows up in two shapes, and both have been seen. Loudly, on the Ingress IP, where the agent names the address in the event, which is the failure #3736 was filed for. Quietly, on the router IP, where the pod comes up and the only symptom is connection refused on a kubelet probe, with ip route get <podIP> on the node resolving to local ... dev lo; that variant is documented in #3445 and #3615. They were being tracked as unrelated flakes in different suites before the shared cause was identified.

What the guard covers, and what it does not

hack/sandbox-cidr-disjoint.bats pins the configuration, not the behaviour. It reads the values out of the tree and fails when they overlap. It does not reproduce the collision, which needs a running cluster.

It checks the invariant rather than the literals, so either value may move as long as they stay disjoint, and it checks the cluster CIDR against the service and join ranges too, because landing on one of those instead is the next way to get this wrong. An extraction that comes up empty, duplicated or misshapen fails the test instead of shrinking the comparison, and comments and blank lines inside the list are skipped rather than read as the end of it. It is not a YAML parser, though: anything it does not read as a bare dotted quad fails the shape validation rather than being understood, which is the safe direction but not the same promise.

Those three are the allocator ranges and nothing more. The sandbox node network, the load-balancer pool and the transit range of a component that is off by default are not read, so the guard is not a complete map of what the cluster CIDR has to avoid. An IPv6 entry fails the shape validation rather than getting an overlap check, so dual-stack means extending the guard rather than passing through it.

Residual risk

I did not prove from source that cilium is the only consumer of Node.spec.podCIDR in this topology. What I did check is that kube-proxy is disabled and that local-ccm runs the node-lifecycle controller only, with no route controller.

This diff selects the full e2e suite, 21 Chainsaw suites with gateway and mongodb among them, so a consumer I missed surfaces as a red run on this PR rather than staying an assumption.

Split of what covers what: the static checks and the unit guard cover the configuration, and the e2e run covers the behaviour. Nothing short of a green suite answers whether cilium's router and Ingress addresses behave once they sit outside the range kube-ovn allocates from.

A check that is not "the suite went green"

hack/e2e-cilium-endpoint-leak-healer.sh already prints a line of its own for this exact collision, so there is a signal to read that does not depend on the overall verdict. In the cozyreport artifact it lands at snapshots/<suite>/host/namespaces/kube-system/v1/pod/cilium-leak-healer-*/healer/current.log, and the lines to grep for are IP is this node's Cilium ingress IP. In the run this was filed from there are 42 of them, all naming the same address.

After this change that line cannot be produced by an address inside the kube-ovn range, because there is no longer an address that is in both. So the expectation is zero from the first run, not zero on average: one occurrence contradicts the change rather than being noise, and is worth stopping on. A green suite can be green for many reasons; this line disappearing has only one.

Scope

This fixes the sandbox and nothing else. For a real installation the cluster CIDR lives in the operator's own machine configuration, outside this repository, so the same collision is still reachable there. #3740 tracks the product side, including the finding that the install documentation does not merely allow the overlap but instructs operators to create it, by requiring podCIDR to match the cluster's pod CIDR.

There is a tradeoff in this worth naming rather than leaving to be discovered. Talos defaults podSubnets to 10.244.0.0/16 (DefaultIPv4PodNet, pkg/machinery/constants/constants.go:888 at v1.13.6, the version this tree ships), and the platform defaults networking.podCIDR to the same value, so an operator who takes both defaults gets the overlap. Until now the sandbox shared that topology and was the one place the bug surfaced where anyone would see it. After this change it no longer reproduces the operator-facing form, and nothing else does either. That is the right call for CI, which should not be wedged by a defect it did not introduce, but it does mean the production exposure loses its only accidental detector.

Closes #3736

To be explicit about what closing that issue does and does not mean: this removes the reproduction in the sandbox and leaves the product configuration untouched. The collision is still reachable for users, and #3740 is where that lives. The issue text describes the root mechanism, which is wider than its title, so the closure covers the e2e failure it was filed for and not the platform default.

Screenshots

Not a UI change.

Downstream repositories

No box is ticked deliberately, and no follow-up PR is opened, so a human decides.

Walking the trigger map against this diff: both hack/e2e-prepare-cluster.bats entries in it are scoped to node prerequisites, the kernel modules a node must load, the containerd settings, the LVM global_filter and the cluster domain. This diff changes none of those, it changes a network range, so no entry fires literally.

The underlying constraint does reach several of them all the same. cozystack/talm sets podSubnets: [10.244.0.0/16] in charts/cozystack/values.yaml, the same range the platform uses for networking.podCIDR, so the documented Talos bootstrap produces the overlap this PR removes from the sandbox. cozystack/ansible-cozystack reaches it from the other side: roles/cozystack/defaults/main.yml sets cozystack_pod_cidr: "10.42.0.0/16" and renders it into networking.podCIDR, while its inventories set cluster-cidr: 10.42.0.0/16, so those two are identical as well. And cozystack/website prescribes the equality outright, install/kubernetes/generic.md stating that podCIDR and serviceCIDR must match the Kubernetes cluster configuration.

No issue is filed there yet on purpose: which side moves is not decided, and asking talm to change podSubnets would be the wrong request if the platform ends up moving instead. That decision belongs to #3740.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

NONE

Summary by CodeRabbit

• Bug Fixes

  • Updated cluster networking ranges to avoid conflicts with Kubernetes, kube-ovn, kilo, and Cilium networks.
  • Added validation to detect missing, invalid, or overlapping cluster and platform CIDR ranges.
  • Preserved safeguards for environments where network ranges overlap, with clearer diagnostic logging.

• Documentation

  • Documented network separation requirements and clarified expected sandbox networking behavior.