You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Opt-in CNI transparent encryption for in-cluster (east-west) traffic #2977
Cozystack ships several networking datapaths as cozystack.networking variants, and the default isp-full (Talos) bundle deploys KubeOVN + Cilium (kubeovn-cilium), not plain Cilium (packages/core/platform/templates/bundles/system.yaml:94). Each datapath has its own in-cluster encryption mechanism, and all are off today:
kubeovn-cilium (default) — KubeOVN can encrypt the Geneve overlay with IPsec, but ENABLE_OVN_IPSEC defaults to false (packages/system/kubeovn/charts/kube-ovn/values.yaml:78).
cilium / cilium-generic — Cilium 1.19.3 supports transparent WireGuard/IPsec encryption, but it is unset in packages/system/cilium/values.yaml.
cilium-kilo — Kilo establishes a WireGuard mesh, but at meshGranularity: location it encrypts only cross-location traffic; pod-to-pod within a location stays in cleartext by design (packages/system/kilo/values.yaml:7).
In every shipped configuration, intra-location / intra-cluster pod-to-pod traffic is unencrypted by default, and there is no single, consistent operator-facing way to turn in-cluster encryption on.
The per-app TLS series (#2814) and Gateway TLS-passthrough (#2815) secure specific, explicitly-exposed endpoints; they do nothing for the bulk of east-west traffic between workloads that do not implement TLS. #2816 mentions CNI mesh encryption only as an optional complement; this issue carves it into an explicit, self-contained opt-in feature.
Goal
A platform-level opt-in that enables in-cluster transport encryption for the active datapath, off by default. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS — defense-in-depth for the whole mesh without per-application changes.
Design questions
Per-datapath knob vs a single platform abstraction. The mechanism differs by variant: KubeOVN ENABLE_OVN_IPSEC, Cilium native WireGuard/IPsec, Kilo WireGuard. A unified networking.encryption.enabled toggle would have to map onto whichever datapath is active.
Default datapath first. Since kubeovn-cilium is the default, the KubeOVN IPsec path is the highest-value target; the Cilium-native path matters for the cilium / cilium-generic variants.
Relationship to Kilo. Kilo already covers cross-location; this feature targets the intra-location gap. Decide whether they compose or are mutually exclusive per variant.
WireGuard vs IPsec trade-offs where both are available; MTU / throughput overhead; interaction with the already-enabled host firewall and kube-proxy replacement.
Non-goals
Replacing application-level TLS or the per-app certificate series (Converge the per-app TLS series onto the ca.crt-only consume contract #2814). In-cluster encryption does not reach an external client and does not give a tenant a verifiable endpoint certificate; it complements end-to-end TLS, it does not substitute for it.
A fully CNI-agnostic abstraction that hides the datapath — the mechanism is necessarily datapath-specific, and surfacing that is acceptable.
Acceptance
A platform value enables in-cluster encryption for at least the default kubeovn-cilium datapath, defaulting to off.
Documentation states which datapath each mechanism covers (incl. Kilo's cross-location-only scope), the measured overhead, and explicitly that this complements rather than replaces application-level / end-to-end TLS.
The feature is verified working on the isp-full (Talos) variant on a live Talos cluster, not by static checks alone: the kube-ovn-cni pod starts cleanly (no CreateContainerError from a read-only-rootfs hostPath), IPsec ESP SAs establish between nodes, and encrypted pod-to-pod traffic flows across nodes. feat(platform): add opt-in in-cluster east-west encryption toggle #2984 is not considered done until this live-on-Talos verification passes.
Heads-up from testing this on a live Talos cluster: with ENABLE_OVN_IPSEC=true (what the networking.encryption.enabled toggle drives on the kubeovn-cilium datapath), the kube-ovn-cni pod fails to start on Talos nodes.
The IPsec path mounts a hostPath for the OVS IPsec keys at /etc/origin/ovs_ipsec_keys. On Talos /etc/origin is a read-only filesystem, so the kubelet can't create the directory:
failed to generate container spec: failed to apply OCI options: failed to mkdir "/etc/origin/ovs_ipsec_keys": mkdir /etc/origin: read-only file system
The container never starts (CreateContainerError), so no IPsec SAs establish (0 ESP SAs) and the overlay datapath is degraded on every Talos node. This affects the isp-full variant, which is the default/primary platform.
Static checks don't surface this — helm-unittest and chart review both pass, because the template renders correctly; the failure is a runtime hostPath-vs-read-only-rootfs issue specific to Talos that only appears on a live Talos node.
Fix direction: the IPsec keys path needs to be Talos-writable — a hostPath under /var (writable on Talos) or an emptyDir — rather than /etc/origin. Holding this in draft until that's addressed.
Context
Cozystack ships several networking datapaths as
cozystack.networkingvariants, and the defaultisp-full(Talos) bundle deploys KubeOVN + Cilium (kubeovn-cilium), not plain Cilium (packages/core/platform/templates/bundles/system.yaml:94). Each datapath has its own in-cluster encryption mechanism, and all are off today:kubeovn-cilium(default) — KubeOVN can encrypt the Geneve overlay with IPsec, butENABLE_OVN_IPSECdefaults tofalse(packages/system/kubeovn/charts/kube-ovn/values.yaml:78).cilium/cilium-generic— Cilium 1.19.3 supports transparent WireGuard/IPsec encryption, but it is unset inpackages/system/cilium/values.yaml.cilium-kilo— Kilo establishes a WireGuard mesh, but atmeshGranularity: locationit encrypts only cross-location traffic; pod-to-pod within a location stays in cleartext by design (packages/system/kilo/values.yaml:7).In every shipped configuration, intra-location / intra-cluster pod-to-pod traffic is unencrypted by default, and there is no single, consistent operator-facing way to turn in-cluster encryption on.
The per-app TLS series (#2814) and Gateway TLS-passthrough (#2815) secure specific, explicitly-exposed endpoints; they do nothing for the bulk of east-west traffic between workloads that do not implement TLS. #2816 mentions CNI mesh encryption only as an optional complement; this issue carves it into an explicit, self-contained opt-in feature.
Goal
A platform-level opt-in that enables in-cluster transport encryption for the active datapath, off by default. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS — defense-in-depth for the whole mesh without per-application changes.
Design questions
ENABLE_OVN_IPSEC, Cilium native WireGuard/IPsec, Kilo WireGuard. A unifiednetworking.encryption.enabledtoggle would have to map onto whichever datapath is active.kubeovn-ciliumis the default, the KubeOVN IPsec path is the highest-value target; the Cilium-native path matters for thecilium/cilium-genericvariants.Non-goals
Acceptance
kubeovn-ciliumdatapath, defaulting to off.isp-full(Talos) variant on a live Talos cluster, not by static checks alone: the kube-ovn-cni pod starts cleanly (no CreateContainerError from a read-only-rootfs hostPath), IPsec ESP SAs establish between nodes, and encrypted pod-to-pod traffic flows across nodes. feat(platform): add opt-in in-cluster east-west encryption toggle #2984 is not considered done until this live-on-Talos verification passes.Part of #2811.