Opt-in CNI transparent encryption for in-cluster (east-west) traffic #2977

Context

Cozystack ships several networking datapaths as cozystack.networking variants, and the default isp-full (Talos) bundle deploys KubeOVN + Cilium (kubeovn-cilium), not plain Cilium (packages/core/platform/templates/bundles/system.yaml:94). Each datapath has its own in-cluster encryption mechanism, and all are off today:

• kubeovn-cilium (default) — KubeOVN can encrypt the Geneve overlay with IPsec, but ENABLE_OVN_IPSEC defaults to false (packages/system/kubeovn/charts/kube-ovn/values.yaml:78).
• cilium / cilium-generic — Cilium 1.19.3 supports transparent WireGuard/IPsec encryption, but it is unset in packages/system/cilium/values.yaml.
• cilium-kilo — Kilo establishes a WireGuard mesh, but at meshGranularity: location it encrypts only cross-location traffic; pod-to-pod within a location stays in cleartext by design (packages/system/kilo/values.yaml:7).

In every shipped configuration, intra-location / intra-cluster pod-to-pod traffic is unencrypted by default, and there is no single, consistent operator-facing way to turn in-cluster encryption on.

The per-app TLS series (#2814) and Gateway TLS-passthrough (#2815) secure specific, explicitly-exposed endpoints; they do nothing for the bulk of east-west traffic between workloads that do not implement TLS. #2816 mentions CNI mesh encryption only as an optional complement; this issue carves it into an explicit, self-contained opt-in feature.

Goal

A platform-level opt-in that enables in-cluster transport encryption for the active datapath, off by default. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS — defense-in-depth for the whole mesh without per-application changes.

Design questions

• Per-datapath knob vs a single platform abstraction. The mechanism differs by variant: KubeOVN ENABLE_OVN_IPSEC, Cilium native WireGuard/IPsec, Kilo WireGuard. A unified networking.encryption.enabled toggle would have to map onto whichever datapath is active.
• Default datapath first. Since kubeovn-cilium is the default, the KubeOVN IPsec path is the highest-value target; the Cilium-native path matters for the cilium / cilium-generic variants.
• Relationship to Kilo. Kilo already covers cross-location; this feature targets the intra-location gap. Decide whether they compose or are mutually exclusive per variant.
• WireGuard vs IPsec trade-offs where both are available; MTU / throughput overhead; interaction with the already-enabled host firewall and kube-proxy replacement.

Non-goals

• Replacing application-level TLS or the per-app certificate series (Converge the per-app TLS series onto the ca.crt-only consume contract #2814). In-cluster encryption does not reach an external client and does not give a tenant a verifiable endpoint certificate; it complements end-to-end TLS, it does not substitute for it.
• A fully CNI-agnostic abstraction that hides the datapath — the mechanism is necessarily datapath-specific, and surfacing that is acceptable.

Acceptance

• A platform value enables in-cluster encryption for at least the default kubeovn-cilium datapath, defaulting to off.
• Documentation states which datapath each mechanism covers (incl. Kilo's cross-location-only scope), the measured overhead, and explicitly that this complements rather than replaces application-level / end-to-end TLS.
• The feature is verified working on the isp-full (Talos) variant on a live Talos cluster, not by static checks alone: the kube-ovn-cni pod starts cleanly (no CreateContainerError from a read-only-rootfs hostPath), IPsec ESP SAs establish between nodes, and encrypted pod-to-pod traffic flows across nodes. feat(platform): add opt-in in-cluster east-west encryption toggle #2984 is not considered done until this live-on-Talos verification passes.

Part of #2811.