docs(platform-package): note the /21 the kube-ovn variants keep for cilium - #739

…not hand out (#4778)

## What this PR does

fixes #3740.

on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
#3740 (comment).

**what changes**

- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.

**existing clusters**

a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.

1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.

running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.

### Validation

- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at cbc9d0c, where #3849 runs
`hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass
after the second commit loads `test_helper`, the strict setup audit is
clean, and helm unittest passes cilium 23/23, platform 263/263 and
kube-ovn 22/22.
- `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real
script against a fake kubectl that models the operator, the agent
daemonset and kube-ovn publishing the subnet status. thirteen mutations
turn it red each (exclusion after the move, no ingress exclusion, no
retry, no status check, an ingress ip inside the block listed on its
own, no `test` op, no wait for kube-ovn, no ip object check, no subnet
filter on the ip objects, no capacity check, a block outside the subnet
let through, overlap turned into containment, no log of ingress ip
holders). the containment one is pinned by a usage range that crosses
the start of the block, `10.244.247.250-10.244.248.3`.
- live on a 3 node talos test cluster with cozystack v1.6.4, cilium
1.19.5 and kube-ovn v1.15.26
- moved from `kubernetes` to the block, exclusion first and then the
node moves this script does. 3 nodes in about 40 seconds, every agent
took its router and ingress ip from its /29, no pod changed its address.
- gateway api through a nodeport 9/9 answered 200 and through the
loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`.
- the final hook job on the moved cluster is a no-op in 2 seconds,
including the ip object read with the job's own rbac. the same ip object
pipeline over that cluster's real objects finds 77 in a populated /24
and none in the block.
- the exact `test` plus `replace` patch on `ovn-default` goes through,
kube-ovn re-sorts the list right after, and the same patch against the
old list is rejected. that's the conflict path the retry covers.
- shellcheck is clean and `hack/check-commit-trailers.sh` passes.
- not run here. the upgrade move from an older release in ci, e2e only
covers the fresh install.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map against the diff.
`packages/core/platform/values.yaml` only gains a comment, but
`networking.podCIDR` gets a new constraint on the kube-ovn variants,
which the hand written key table on the website should say. the ansible
role sets `networking.podCIDR` to `10.42.0.0/16`, which yields
`10.42.248.0/21` and needs nothing there. the talm preset keeps
`podSubnets: 10.244.0.0/16`, which no longer matters on these variants
because cilium stops reading `Node.spec.podCIDR` there. the terraform
provider doesn't model platform values.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#739
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR
for Cilium router and Gateway API Ingress addresses, excluding it from
Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20`
or shorter.
* Added a post-upgrade migration that moves existing Cilium IP
allocations into the cluster pool and updates affected node networking.
* **Documentation**
* Documented the pod CIDR requirements and node capacity limit for
Kube-OVN variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->