feat(platform): add opt-in in-cluster east-west encryption toggle - #2984

No actionable comments were generated in the recent review. 🎉

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements an opt-in feature for in-cluster network encryption, providing a defense-in-depth layer for pod-to-pod traffic. By enabling IPsec for the KubeOVN Geneve overlay, the change secures inter-node communication without requiring application-level modifications. The implementation is designed to be backward compatible, off by default, and includes detailed documentation regarding its scope, performance implications, and limitations.

Highlights

• In-cluster Encryption Toggle: Introduced a new configuration option networking.encryption.enabled in the platform chart to enable in-cluster east-west transport encryption.
• KubeOVN IPsec Integration: Mapped the new toggle to KubeOVN's overlay IPsec (ENABLE_OVN_IPSEC), leveraging KubeOVN's native controller for certificate management via the Kubernetes CSR API.
• Validation and Testing: Added comprehensive helm unittest coverage to ensure the toggle correctly handles various states and preserves existing KubeOVN overrides.

Heads-up from testing this on a live Talos cluster: with ENABLE_OVN_IPSEC=true (what the networking.encryption.enabled toggle drives on the kubeovn-cilium datapath), the kube-ovn-cni pod fails to start on Talos nodes.

The IPsec path mounts a hostPath for the OVS IPsec keys at /etc/origin/ovs_ipsec_keys. On Talos /etc/origin is a read-only filesystem, so the kubelet can't create the directory:

failed to generate container spec: failed to apply OCI options: failed to mkdir "/etc/origin/ovs_ipsec_keys": mkdir /etc/origin: read-only file system

The container never starts (CreateContainerError), so no IPsec SAs establish (0 ESP SAs) and the overlay datapath is degraded on every Talos node. This affects the isp-full variant, which is the default/primary platform.

Static checks don't surface this — helm-unittest and chart review both pass, because the template renders correctly; the failure is a runtime hostPath-vs-read-only-rootfs issue specific to Talos that only appears on a live Talos node.

Fix direction: the IPsec keys path needs to be Talos-writable — a hostPath under /var (writable on Talos) or an emptyDir — rather than /etc/origin. Holding this in draft until that's addressed.

Follow-up from the same Talos cluster, with the keys path fixed: the direction above was right and the knob exists. OVN_IPSEC_KEY_DIR is a kube-ovn value, and we already redirect its siblings (OPENVSWITCH_DIR, OVN_DIR) to /var/lib but never set the IPsec one, so it fell back to the read-only /etc/origin default. Setting OVN_IPSEC_KEY_DIR: /var/lib/ovs_ipsec_keys makes kube-ovn-cni start cleanly.

That half is really an upstream documentation gap, and I sent it there: the Talos sections of both kube-ovn charts list only two of the three host directories and leave the IPsec one at its /etc/origin default, so following the documented instructions and then enabling IPsec reproduces the error above. kubeovn/kube-ovn#7041 adds the missing directory to both sections, together with a render fix it turned up along the way — the ovs-ipsec-keys volume is indented one level too deep, which makes the v2 chart fail to render at all whenever enableOvnIpsec is set.

That unblocks container creation, but not the feature. With it applied IPsec does come up — 8 ESP SAs establish between node IPs in transport mode, and notably they establish with cilium.hostFirewall.enabled=true, so the host firewall does not block SA establishment. Two blockers remain, and both are why this stays in draft:

1. Cross-node pod-to-pod traffic is 100% lost while IPsec is on. The SAs are up, but the encrypted datapath carries nothing. I have not isolated it further; the candidates are ESP (IP proto 50) dropped on the underlay, MTU/fragmentation, or a kube-ovn IPsec datapath bug on this setup. This is distinct from SA establishment, which demonstrably works.
2. There is no clean teardown. Turning IPsec back off leaves stale xfrm SAs and policies behind — kube-ovn does not flush them on disable — so cross-node pod-to-pod stays broken after the rollback until ip xfrm state flush and ip xfrm policy flush are run by hand on every node. That makes the toggle a one-way door in practice.

So the blocker is neither the chart wiring nor the keys path — both are solved. It is that kube-ovn IPsec does not carry traffic on this datapath and cannot be turned off cleanly, which looks like an upstream problem rather than something this toggle can paper over. Unlike the keys path, these two are not reported upstream yet. I am keeping this in draft until the datapath and the teardown are fixed: shipping an opt-in that silently blackholes pod-to-pod traffic and cannot be reverted without manual surgery on every node is not worth the defense-in-depth.