bug(cozy-proxy): WholeIP 1:1 NAT broken with Cilium 1.19 due to wildcard service drop entries #2327

Description

After upgrading Cozystack from v0.41.x (Cilium 1.18.6) to v1.2.1 (Cilium 1.19.1), VMs configured with externalMethod: WholeIP lose connectivity on ports not explicitly listed in externalPorts.

The cozy-proxy component implements stateless 1:1 NAT using nftables raw/prerouting rules for WholeIP services. This worked correctly with Cilium 1.18.6, but breaks with Cilium 1.19.1 due to a change in BPF LB behavior.

Root Cause

Cilium 1.19 introduced wildcard service drop entries in PR cilium/cilium#40684 (commit 9f67ad4471):

"The Cilium datapath will now drop TCP/UDP traffic towards a LoadBalancer or ClusterIP allocated by LB-IPAM in the north-south direction if the destination port does not match a provisioned service."

How it works

When a LoadBalancer Service is created, Cilium now programs a wildcard BPF LB entry:

<LB_IP>:0/ANY → 0.0.0.0:0 [LoadBalancer, non-routable]

When a packet arrives for a port not in the Service spec:

1. BPF does lookup for <LB_IP>:<port> → MISS
2. BPF does wildcard lookup for <LB_IP>:0/ANY → HIT (non-routable)
3. Packet is dropped with DROP_NO_SERVICE before reaching netfilter

Since cozy-proxy relies on nftables raw/prerouting (which runs after BPF TC), it never sees the packet.

Cilium 1.18 behavior (working)

Packet → BPF LB (port not found) → pass to host stack → nftables raw → 1:1 NAT → VM

Cilium 1.19 behavior (broken)

Packet → BPF LB (port not found) → wildcard lookup → DROP_NO_SERVICE

Reproduction

1. Create a VM with externalMethod: WholeIP and externalPorts: [22]
2. The LoadBalancer Service only contains port 22
3. The VM's OPNsense web UI on port 443 is unreachable externally
4. Adding port 443 to the Service (or externalPorts) fixes it
5. Downgrading Cilium to 1.18.6 also fixes it

Suggested Fix

For VMs with externalMethod: WholeIP, cozy-proxy's 1:1 NAT should handle all ports. Possible approaches:

1. vm-instance chart: When externalMethod: WholeIP, do not create a LoadBalancer Service at all (let cozy-proxy handle everything via nftables without Cilium interference)
2. vm-instance chart: When externalMethod: WholeIP, create a headless Service or use a Cilium annotation to skip BPF LB for this IP
3. cozy-proxy: Move from nftables to eBPF-based NAT that hooks earlier in the datapath (before Cilium TC)
4. Pin Cilium: Document that Cilium 1.18.x is required when using WholeIP (temporary workaround)

Environment

• Cozystack v1.2.1
• Cilium 1.19.1 (broken), Cilium 1.18.6 (works)
• VMs: OPNsense with externalMethod: WholeIP