Migrate LoadBalancer announcer from MetalLB to Cilium native (LB-IPAM + L2) #2586

Summary

Replace MetalLB with Cilium-native LoadBalancer announcement (LB-IPAM for IP allocation, CiliumL2AnnouncementPolicy for ARP/NDP). Drop the packages/system/metallb/ chart once the migration is complete. The IP-allocation half is already in flight on feat/ingress-lb-ipam; this issue tracks the announcer half and the operator-facing migration story.

Current state

• MetalLB v0.15.2 in packages/system/metallb/, L2 mode, no BGP configuration.
• Cilium 1.19.3 in packages/system/cilium/charts/cilium/ with enableLBIPAM: true already on by default.
• CiliumLoadBalancerIPPool is rendered by the tenant chart in packages/apps/tenant/templates/cilium-lb-pool.yaml.
• Branch feat/ingress-lb-ipam switches publishing.exposure from the deprecated Service.spec.externalIPs (removal scheduled around k8s v1.43) to loadBalancer mode, so IP allocation already flows through Cilium LB-IPAM. MetalLB remains as the L2 announcer only.

Motivation

• Removes a second LoadBalancer controller from the platform: one chart, one CRD set, one DaemonSet less to maintain and update.
• Removes the soft dependency on MetalLB's lifecycle (chart updates, CRD migrations) for a feature that the CNI already provides natively.
• Aligns with Cilium's own deprecation of the MetalLB-derived BGP v1 path (CiliumBGPPeeringPolicy) in favor of native announcement primitives.
• Unblocks consistent annotations across the platform (lbipam.cilium.io/* instead of mixed metallb.universe.tf/* + lbipam.cilium.io/*).

Why both can coexist on the dataplane

This matters for the migration mechanism, so worth being explicit. Both MetalLB-L2 and Cilium L2 are pure userspace ARP/NDP responders. Neither binds the LoadBalancer IP to an interface, neither installs iptables or eBPF rules for the ingress traffic. The actual LB datapath is Cilium eBPF on every node (kube-proxy-replacement is a hard prereq for Cilium L2 anyway, and is already on). Whoever answered the ARP request only decides which node receives the packet on the wire; once it arrives, eBPF forwards it to a backend pod regardless. This means the announcer can be swapped without touching the dataplane and without a real connection-level outage — only an ARP cache propagation window on the upstream router.

Proposal — phased

Phase 0 — prerequisite

Land feat/ingress-lb-ipam. After this, every LoadBalancer Service receives its IP from a CiliumLoadBalancerIPPool. MetalLB is no longer the IPAM source, only the announcer. This is independent and useful on its own (it removes the externalIPs dependency).

Phase 1 — opt-in Cilium announcer

Introduce a platform-level switch:

# packages/core/platform/values.yaml
networking:
  loadBalancerAnnouncer: metallb  # metallb | cilium

Behavior:

• metallb (default): MetalLB chart is installed, cilium.l2announcements.enabled=false. Current behavior, no change for existing clusters.
• cilium: MetalLB chart is not installed (or is uninstalled on switch), cilium.l2announcements.enabled=true, a CiliumL2AnnouncementPolicy is rendered to cover the same node and Service set MetalLB used to.

Mutually exclusive by construction — no risk of both controllers being active simultaneously.

Documented as experimental in this phase. Release notes call out the migration steps, the ARP-propagation window, and the known limitations below.

Phase 2 — flip the default

After at least one bake-in cycle on real clusters, change the default to loadBalancerAnnouncer: cilium. MetalLB remains installable for clusters that need to delay the switch.

Phase 3 — drop MetalLB

Remove packages/system/metallb/, the corresponding PackageSource, the bundle entry, and the loadBalancerAnnouncer flag itself.

Migration mechanism for the operator

The switch is a sequential cutover, not a parallel run:

1. Operator changes loadBalancerAnnouncer from metallb to cilium and applies platform values.
2. Installer/controller scales MetalLB speakers to zero (or uninstalls the HelmRelease). MetalLB stops responding to ARP and stops sending gratuitous ARP. Existing TCP connections on every LoadBalancer IP keep working: the upstream router still has the old MAC cached, packets still arrive at the previous leader node, eBPF on that node still routes to backend pods.
3. Installer applies the CiliumL2AnnouncementPolicy. Cilium agents elect a leader per IP (independent from MetalLB's prior election — may be the same node, may be a different one) and send gratuitous ARP. The upstream router updates its cache. From the client's perspective, no flow is broken.

Brief edge cases:

• Clients with no ARP cache for the IP (cold) and arriving in the gap before Cilium sends its first gratuitous ARP — they will ARP the network, get no reply for a few seconds, retry, succeed once Cilium answers.
• Stateful UDP flows (notably DNS over UDP for long-running clients) where the upstream router's MAC actually changes mid-flow — the conntrack on the new leader node will treat the first packet as a new flow. For typical ingress workloads this is invisible.
• If the operator is cautious, draining one Service at a time is also possible by setting loadBalancerClass per Service, but this adds Helm-template complexity for limited gain.

Known limitations

• Cilium L2 is incompatible with Service.externalTrafficPolicy: Local. MetalLB-L2 has the same limitation by construction (only one node responds to ARP), so this is not a regression — but it's worth documenting against the platform's ingress chart.
• Sharing one IP across Services with different pod selectors is gated behind an opt-in annotation in upstream Cilium (cilium/cilium#45819, tracking cilium/cilium#38766). Once that lands, the annotation is lbipam.cilium.io/sharing-permit-different-pods and must be set on both Services in the sharing group. Not used by any in-tree cozystack chart today; flagged here so the platform can pick it up if a tenant scenario starts depending on it.
• Lease-based leader election uses the apiserver. At cozystack-typical scale (tens of LoadBalancer IPs per cluster) this is negligible; flagging it because it's a real architectural difference vs MetalLB's gossip.
• Avoid Cilium versions with active L2 / LB-IPAM regressions on the upgrade path: 1.19.0 / 1.19.1 had several. Pin to a stable 1.18.x or 1.19.2+ for the cutover.

Acceptance criteria

• feat/ingress-lb-ipam merged.
• networking.loadBalancerAnnouncer flag added to packages/core/platform/values.yaml with schema validation.
• cilium mode installs CiliumL2AnnouncementPolicy covering the same selectors MetalLB used.
• cilium mode ensures MetalLB chart is uninstalled (no co-existence in the same cluster).
• Existing chart annotations referring to metallb.universe.tf/* are translated to lbipam.cilium.io/* equivalents in the tree.
• e2e test exercises the switch and verifies LoadBalancer IPs remain reachable across the transition.
• Migration documentation in docs/operations/ covering the cutover steps and the known limitations.
• Default flipped to cilium (Phase 2) — separate PR.
• MetalLB chart removed (Phase 3) — separate PR.

Out of scope

• BGP announcement. The default cozystack LoadBalancer story is L2; BGP can be added separately as another value of loadBalancerAnnouncer if/when there's user demand.
• Gateway API integration. Tracked separately; the LB-IPAM + Gateway combination has had a known-bad interaction recently upstream and shouldn't be conflated with this change.