CFP: Allow shared lb ipam ip on services selecting different pods #38766
Description
Cilium Feature Proposal
Hello, while setting up a new kubernetes deployment on a vps provider which only provides a single public ip. I was hoping to use lb ipam to satisfy the LoadBalancer service ip allocation.
This worked flawlessly for the first service which referenced the ip pool. However when I attempted to share the ip with another service I continuously received the out_of_ips reason on the second service.
After digging into ServiceView.isCompatible it appears there is a limitation placed which disallows two services using the externalTrafficPolicy of Local to select a different set of pods.
The following comment was provided for the reason for that limitation:
If both services have a 'local' external traffic policy, then they must select the same set of pods.
If this were not the case, then we could end up in a situation directing traffic to a node which doesn't
have the pod running on it for one of the services.
This however doesn't allow a user who is aware of this routing limitation and handles such cases, to use the lb ipam provider.
In my case, I have two services deployed as kind: DaemonSet ensuring that both services exist on all nodes.
So I'm proposing either dropping this limitation or providing a way to bypass this check.
If this is something Cilium would be open to, I'm happy to give it a first stab once we settle on the proper path forward.
Greatly appreciate this project and look forward to contributing to it!
Activity
lb-ipam: allow local services sharing ips to reference different pods
lb-ipam: allow local services sharing ips to reference different pods
dylandreimerink commentedon Apr 15, 2025
I would be better if we can actually assert that the two services have backends on the same set of nodes. This should be theoretically possible, but a bit challenging to implement in a efficient way. Just disabling the check is going to cause some insidious bugs for unsuspecting users.
Thinking aloud on how to go about that.
The naive approach would be to query all endpoints selected by both services (via its selector or endpointslice), taking the list of nodes and comparing them. This requires querying and watching the endpoints. The main question being, what do we do if at some point pods get rescheduled such that they no longer live on the same nodes. Just removing the allocation from one of the services will break it.
I think this might be the only way to do it in a way where chances of disruption due to endpoint churn and the like is not an issue. What we could do is still resolve the backends for a service, but assert that all of them must be part of a
DaemonSet. We do not check the actual list of nodes, but make the assumption that if both services only select deamonset pods, that the set of nodes is eventually consistent.We would also have to ensure the node selector and node affinity of all DaemonSets match.
This should allow your use case, and provide more safeguards without making it to fragile. WDYT?
mikemrm commentedon Apr 27, 2025
Sorry for the late response, I missed the notification.
I think having a way to override the check with an annotation on the service would be the simplest and most versatile solution while still keeping default checks. This would allow users to opt-out of these guide rails with something like
lbipam.cilium.io/sharing-permit-different-pods: 'true'on all services sharing the same key.lb-ipam: allow local services sharing ips to reference different pods
dylandreimerink commentedon Apr 28, 2025
I am not discounting that solution outright. But just making an annotation to disable checks feels like the "easy way" out of the issue. At the core, you are doing the right thing, and the software isn't smart enough to recognize that. So why not make the software smarter instead of making a "i know better" annotation?
mikemrm commentedon Apr 29, 2025
I agree, making the software smarter is probably better in the long run. I'm not currently familiar with kubernetes apis, so I'm unfamiliar with what would be necessary to achieve this.
What would be a good next step for this?
haraldkoch commentedon Jul 2, 2025
I think I just discovered this limitation while migrating (working) Prometheus services from a Calico-based cluster, attempting to use "Sharing Keys" as described here: https://docs.cilium.io/en/latest/network/lb-ipam/#sharing-keys
This works for Prometheus exporters because each one runs on a different service port. Does Cilium's traffic routing not take into account the ports declared in the service? If the ports are unique, then the [ip,port] tuple is also unique, and there should be no conflict.
I note that the documentation for Sharing Keys does mention services with unique ports, and does not mention any
ExternalTrafficPolicy: Locallimitation:github-actions commentedon Sep 1, 2025
This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
github-actions commentedon Sep 16, 2025
This issue has not seen any activity since it was marked stale.
Closing.
12 remaining items
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
revert(fenrir): back out externalTrafficPolicy: Local for dreamcast
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods
lbipam: allow sharing IPs across services that select different pods