Repository navigation
Make Cilium BGP control plane a first-class LoadBalancer path for Cilium clusters #2804
Description
Summary
On Cilium-based clusters, advertising LoadBalancer / externalIP service IPs over BGP via Cilium's own BGP control plane is cleaner than MetalLB — advertisement and the BPF datapath live in one place, with no MetalLB↔Cilium handoff to misalign. Cozystack technically supports this (the vendored Cilium chart exposes bgpControlPlane), but it's entirely manual and undocumented, and MetalLB is the default LB. The Cilium-BGP path should be made first-class: documented, and ideally toggleable.
Current state
- MetalLB ships as a default system package (
cozystack.metallb, pulled into the common system bundle viapackage.default) and is the out-of-the-box LoadBalancer implementation. - The vendored Cilium chart supports
bgpControlPlane(packages/system/cilium/charts/cilium/values.yaml, defaultfalse). Cilium's operator registers theCiliumBGP*CRDs at runtime when it's enabled. - Cozystack does not pin
cilium.devices— Cilium auto-detects, so on multi-NIC nodes the BPF LB datapath may not attach to the interface that carries the BGP session, and north-south LB traffic loops back to the upstream router.
What an operator has to do today (manual, undocumented)
- Disable MetalLB:
bundles.disabledPackages: [cozystack.metallb]. - Enable Cilium BGP via the Cilium Package:
cilium.bgpControlPlane.enabled: true. - Pin
cilium.devicesto include the BGP-facing NIC, otherwise the LB IP loops (Cilium does not add a host route by design — the BPF datapath must catch the packet on a managed device). - Apply
CiliumBGPClusterConfig/CiliumBGPPeerConfig/CiliumBGPAdvertisementCRs by hand.
Proposal
- Document the Cilium-BGP LoadBalancer path in the networking docs: the MetalLB mutual-exclusion, the
cilium.devicesrequirement, and a working set of CiliumBGP CRs. - Optionally expose it as a supported toggle (e.g. a
publishing-level switch or a Cilium Package variant) so operators don't hand-assemble it and MetalLB is disabled automatically when it's selected.
Background
Surfaced from a user exposing tenant-root ingress-nginx over BGP: switching from MetalLB to the Cilium BGP control plane advertised the IP correctly, but traffic looped — Cilium doesn't add a host route (by design) and the BPF datapath wasn't attached to the BGP NIC because cilium.devices was unset.
Activity
lexfrei commentedon Jun 3, 2026
Independently corroborated from a separate field deployment (L3 + FRR): Cilium's external LoadBalancer datapath did not work until the ethernet interfaces were pinned explicitly (
cilium.devices: 'eno5np0 eno6np1 dummy0 lo'). Same root cause as this issue — auto-detection misses the NIC carrying the LB traffic, so the BPF LB program isn't attached there. Reinforces pinningcilium.devicesas part of the first-class BGP-LB path.lexfrei commentedon Jun 4, 2026
Follow-up from the same thread: once the Cilium-BGP path is working, a recurring source of confusion is that
traceroute/ping/ ICMP to the LoadBalancer VIP shows a routing loop even though the service is fully reachable. This is expected — Cilium's BPF LB only DNATs packets that match a service L4 backend (e.g. TCP :443), so ICMP and non-service ports fall through to the kernel's default route and bounce back to the upstream router. It has zero effect on real traffic. The networking docs this issue calls for should include that note explicitly, and recommend monitoring the service port (a TCP connect probe) rather than an ICMP ping of the VIP — it's a cosmetic symptom that reads as a misconfiguration and costs hours to chase down.