[Epic] Cilium: tracking version pins, Gateway API, network policy, IPAM and LB issues #3913

Tracking issue for everything where Cilium is the subject: the pinned version and its bumps, the Gateway API implementation, network policy, IPAM and the datapath next to kube-ovn, LB-IPAM and BGP, and the diagnostics around them. Every item below also carries area/cilium, so is:open label:area/cilium is the live list; this issue is the map and the place for cross-cutting notes. Add new Cilium items here when filing them.

Where we stand

packages/system/cilium vendors chart 1.19.5 and builds the first-party agent image from quay.io/cilium/cilium:v1.19.5 (packages/system/cilium/images/cilium/Dockerfile); the operator image comes from the vendored chart. The agent image is digest-pinned in packages/system/cilium/values.yaml and is not repinned by a chart bump. One BPF patch is carried (IPv6 passthrough to the kernel stack when the IPv6 datapath is disabled under host firewall).

The bump to v1.19.6 (#3343) is in draft: v1.19.6 and v1.19.7 reject the default TenantGateway listener layout, see #3912 for the mechanism and the options. v1.19.6 also carries the fix for #3721 (cilium/cilium#42159), which is the reason the bump matters.

Version and bump

• fix(cilium): bump vendored chart to v1.20.2 #3343 bump vendored chart to v1.19.6 (draft, blocked on Cilium v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; the Cilium bump is blocked on a listener layout decision #3912)
• Cilium v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; the Cilium bump is blocked on a listener layout decision #3912 v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; listener layout decision
• Revert TenantGateway port-443 allowedRoutes.kinds workaround once Cilium ≥1.20 lands #3087 revert the TenantGateway port-443 allowedRoutes.kinds workaround once Cilium ≥1.20 lands
• bug(cozy-proxy): WholeIP 1:1 NAT broken with Cilium 1.19 due to wildcard service drop entries #2327 cozy-proxy WholeIP 1:1 NAT broken with Cilium 1.19 (wildcard service drop entries)

Upstream requests we opened

• gateway-api: 1.19.6 regression: same-port HTTPS and TLS passthrough listeners rejected on any SNI overlap, wildcard next to exact included cilium/cilium#48092 narrow the same-port HTTPS/TLS-passthrough conflict to equal hostnames, apply exact-before-wildcard for a wildcard superset (the rule behind Cilium v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; the Cilium bump is blocked on a listener layout decision #3912); offered to do the PR

Gateway API on Cilium

• dashboard.xxx.xxx HTTPRoute .status.parents reports HTTPRoute is not allowed to attach to this Gateway due to namespace selector restrictions #3721 platform HTTPRoutes rejected with NotAllowedByListeners on a correctly labelled namespace (Gateway API: allowedRoutes is incorrectly applied to all listeners when sectionName is omitted cilium/cilium#42159, fixed in v1.19.6)
• api.xxx.xxx HTTPS and TLS conflicted in the tenant-root's TenantGateway cozystack listeners #3720 api.<apex> HTTPS terminate and TLS passthrough under one SNI wedge the Envoy listener on http01 (fix in flight: feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342)
• dns01 gateway renders a terminate and a passthrough listener under one SNI on the pinned Cilium #3718 dns01/existingSecret render a terminate and a passthrough listener under one SNI on the pinned Cilium
• feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342 TLS-passthrough listeners on TenantGateway (PR)
• Gateway API (Cilium): no supported knob to enable proxyProtocol on the publishing path #3030 no supported knob to enable proxyProtocol on the publishing path
• External database exposure via Gateway API TLS-passthrough (SNI) #2815 external database exposure via Gateway API TLS passthrough (SNI)
• Gateway API + LB-IPAM: cilium-gateway Service LoadBalancer IP flaps (release/re-assign ~1/s), breaking BGP service advertisement (v1.5.2) #3307 cilium-gateway Service LoadBalancer IP flaps under LB-IPAM, breaking BGP advertisement
• test(e2e): end-to-end traversal test for the Kubernetes API TLSRoute #2478, e2e: the TLSRoute half of the route-hostname policy has no runtime probe #3871 no e2e coverage for the Kubernetes API TLSRoute and the TLSRoute half of the hostname policy (why Cilium v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; the Cilium bump is blocked on a listener layout decision #3912 is invisible to CI)

Network policy

• apps/tenant renders Cilium policies unguarded, breaking installs without Cilium (isp-hosted) #3387 apps/tenant renders Cilium policies unguarded, breaking installs without Cilium
• OIDC access fails for sub-tenant clusters due to missing Cilium egress rule #3486 OIDC access fails for sub-tenant clusters, missing Cilium egress rule
• tenant: allow-to-apiserver only opens port 6443, breaks in-cluster clients using :443 #2773 tenant allow-to-apiserver only opens 6443, in-cluster clients on :443 break
• Nested cluster Fluent Bit cannot send logs to parent VLogs: missing vlogs in tenant egress policy #1970 nested cluster Fluent Bit cannot reach parent VLogs, missing vlogs in the tenant egress policy
• toFQDNs policies have no effect on the default kube-ovn + Cilium stack, and nothing documents that #3820 toFQDNs policies have no effect on the default kube-ovn + Cilium stack, and nothing documents that

IPAM and datapath next to kube-ovn

• networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740 Cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from
• e2e: pod IP becomes a local address of the node host netns, kubelet probes its own loopback #3615 e2e: pod IP becomes a local address of the node host netns, kubelet probes its own loopback (same two-allocators-one-CIDR shape as networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740)
• restrict-system-components blocks LINSTOR satellite cross-node traffic when Cilium is chained with kube-ovn #3337 restrict-system-components blocks LINSTOR satellite cross-node traffic when Cilium is chained with kube-ovn

LB-IPAM, L2, BGP

• Make Cilium BGP control plane a first-class LoadBalancer path for Cilium clusters #2804 make the Cilium BGP control plane a first-class LoadBalancer path
• feat(vpn): move the vpn app off Service.spec.externalIPs onto CiliumLoadBalancerIPPool #2477 move the vpn app off Service.spec.externalIPs onto CiliumLoadBalancerIPPool
• Gateway API + LB-IPAM: cilium-gateway Service LoadBalancer IP flaps (release/re-assign ~1/s), breaking BGP service advertisement (v1.5.2) #3307 (also above) LB-IPAM IP flaps

Docs and diagnostics

• docs: Hubble enablement snippet doesn't say where to put it (Package values nesting) #2772 Hubble enablement snippet does not say where to put it
• test(e2e): cilium-dbg bpf lb list runs only when a LoadBalancer Service is broken, so a ClusterIP failure never gets the one table that would explain it #3689 e2e: cilium-dbg bpf lb list runs only when a LoadBalancer Service is broken

Adjacent, not labelled

• E2E kubernetes-previous fails consistently: Cilium install times out inside tenant cluster on v1.34.9 nodes #3229 kubernetes-previous Cilium install timeout inside the tenant cluster; E2E: tenant worker bring-up misses the node-Ready budget with zero Nodes registered — survived #3455's fsync fix and the 12m→18m raise #3513 reads it as a symptom of zero tenant nodes registering, not a Cilium fault
• gateway: a second tenant Gateway steals the platform namespaces from the first #3869, gateway: the documented namespace-selector guarantee does not describe the selector the chart renders #3867, gateway: route Accepted status is decided by the hostname race alone, and status write failures are swallowed #3865, TenantGateway reports Accepted=True for a route whose sectionName names no listener #3764 TenantGateway controller behaviour on top of Cilium; they are about our controller, not Cilium, and stay under area/networking