Repository navigation
Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster #1563
Description
Describe the bug
Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster:
Example:
---
apiVersion: v1
kind: Service
metadata:
name: lb-test1
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
selector:
app: app-test1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-test1
spec:
replicas: 1
selector:
matchLabels:
app: app-test1
template:
metadata:
labels:
app: app-test1
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
These IPs have already been used in previous versions of CozyStack. If I create multiple LBs, and the pool will issue IPs that have not been used before, then everything is fine.
Debugging shows that metallb complains that there are no Endpoints, although there are EPs inside the tenant cluster, and if I do port forwarding to svc LB inside tenant cluster, everything works:
$ kc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
lb-test1 LoadBalancer 10.95.164.25 2xx.xxx.xxx.xxx 80:32479/TCP 42s
$ kc port-forward svc/lb-test1 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
IPs that are successfully announced have an endpoint. This endpoint is the IP address of virt-launcher pod in the root cluster, i.e., the node of the tenant k8s. And the NodePort is the external port of that root k8s node where virt-launcher is located. This NodePort is open externally if you try to connect to it on the IP node of root k8s.
Also, metallb-controller and metalb-speaker respond, and there are entries in Events.
# kc describe svc a375e7eaa3e994966b8920136c013b45 -n tenant-qwerty
Name: a375e7eaa3e994966b8920136c013b45
=== cut ===
Type: LoadBalancer
TargetPort: 30815/TCP
NodePort: <unset> 32611/TCP
Endpoints: 10.244.3.155:30815
External Traffic Policy: Cluster
Internal Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 9m38s metallb-controller Assigned IP ["2xx.xxx.xxx.xxx"]
Normal nodeAssigned 9m38s metallb-speaker announcing from node "qwerty-xxx-xxx-xxx-1" with protocol "layer2"
# nc -v 2xx.xxx.xxx.xxx 32611
(UNKNOWN) [2xx.xxx.xxx.xxx] 32611 (?) open
The external node port of the problematic svc is unavailable, and the speakers have no records:
# kc describe svc -n tenant-qwerty a5ed58225d6b941e3885fe37d59801d2
Name: a5ed58225d6b941e3885fe37d59801d2
=== cut ===
Type: LoadBalancer
Port: <unset> 80/TCP
TargetPort: 32479/TCP
NodePort: <unset> 31985/TCP
Endpoints: <none>
External Traffic Policy: Cluster
Internal Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 46m metallb-controller Assigned IP ["2xx.xxx.xxx.yyy"]
# nc -v 2xx.xxx.xxx.zzz 31985
(UNKNOWN) [2xx.xxx.xxx.zzz] 31985 (?) : Connection refused
And in the logs, there is a message saying that there are no endpoints.
{"caller":"layer2_controller.go:84","event":"shouldannounce","ips":["2xx.xxx.xxx.yyy"],"level":"debug","message":"failed no active endpoints","pool":"cozystack","protocol":"l2","service":"tenant-qwerty/a5ed58225d6b941e3885fe37d59801d2","ts":"2025-10-31T14:45:43Z"}
Please tell me what else I can check?
Environment
- Cozystack version: 0.37.2
- Provider: on-prem
Activity
dosubot commentedon Oct 31, 2025
This issue is likely related to how endpoints are propagated from tenant clusters to the root cluster, especially for IPs that were previously used in older CozyStack versions. There's a known bug where a panic in the kubevirt cloud controller manager prevents endpoints from being correctly created in the root cluster, which causes MetalLB to report "no active endpoints" and not announce the IP externally—even though everything works inside the tenant cluster. This matches your symptoms and has been discussed in detail here.
To troubleshoot, check for errors or panics in the kubevirt cloud controller manager logs in the root cluster. If you see signs of a crash or nil pointer dereference, this is almost certainly the cause. The fix for this was prepared upstream and merged into CozyStack (PR link), so make sure your environment includes this patch.
If you suspect stale resources, you can manually check for leftover endpoints, NodePorts, or IP allocations in both clusters. Use
kubectl get endpoints -Aandkubectl get svc -Ain both root and tenant clusters to look for services with missing endpoints or NodePorts. If you find any, try deleting and recreating the affected LoadBalancer service to force a fresh allocation and endpoint mapping.Also, confirm that your MetalLB IP pool only contains unused IPs and that routing is correct for all IPs in the pool (docs). Sometimes, previously used IPs may be blocked by lingering resources or misconfigured routes.
If the issue persists after cleanup and patching, review the events and logs for MetalLB and kubevirt controllers for clues about resource propagation failures.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
[-]Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster[/-][+]Some IPs in the pool for MetalLB are not available when creating svc `LoadBalancer` in a tenant cluster[/+]kinseii commentedon Oct 31, 2025
Thanks to the bot, I checked the KCCM log and found a multiple svc error. However, I don't have multiple svc, so these are probably old entries in etcd.
dosubot commentedon Oct 31, 2025
The "multiple services found for tenant endpoint slice" error in the CCM logs happens when the controller detects more than one Service resource associated with the same EndpointSlice in the infra cluster. This usually means there are stale or orphaned Service entries lingering in etcd or the infra cluster—often from previous CozyStack versions or failed deletions—even if they're not visible with standard kubectl commands. These hidden resources can block correct endpoint mapping and break MetalLB LoadBalancer provisioning, especially for IPs reused from older clusters or services.
There isn't a built-in automated cleanup for this in CozyStack. The recommended approach is to manually identify and remove any leftover Service or EndpointSlice resources in the infra cluster's etcd. You can try:
kubectl get svc -Aandkubectl get endpointslices -Ain both tenant and infra clusters.etcdctldirectly on the infra cluster's etcd to search for and remove orphaned Service or EndpointSlice keys.This issue is more likely to occur with IPs that were previously allocated, so cleaning up these stale resources should resolve the mapping conflict and restore MetalLB functionality. For more background, see the discussion and upstream fix here: cozystack/cozystack#1017.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
kinseii commentedon Oct 31, 2025
# (@dosu) How can I determine that endpointslice has multiple svc?
dosubot commentedon Oct 31, 2025
The CCM error "multiple services found for tenant endpoint slice" means the controller found more than one Service resource associated with a single EndpointSlice—usually due to stale or orphaned resources in etcd that aren't visible with normal kubectl commands.
To check for this:
List all EndpointSlices and Services in the affected namespace with:
For a specific EndpointSlice (e.g.,
lb-test1-nxzlk), inspect its labels and ownerReferences:Look for labels like
kubernetes.io/service-nameor ownerReferences pointing to a Service.List all Services and check if more than one has a matching label or selector for that EndpointSlice. For example:
If you see more than one Service referencing the same EndpointSlice, that's the conflict.
If you suspect hidden resources, you may need to use
etcdctlon the infra cluster to search for duplicate Service or EndpointSlice keys. The CCM log itself is a strong indicator—if you see this error, there are definitely multiple Service objects (even if not visible via kubectl) associated with the EndpointSlice in etcd or the infra cluster. Cleaning up these stale resources should resolve the issue.For more context, see the discussion in cozystack/cozystack#1017.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
kinseii commentedon Oct 31, 2025
# (@dosu) how to find duplicates through etcdctl
dosubot commentedon Oct 31, 2025
To find duplicate or orphaned Service and EndpointSlice resources in etcd using etcdctl (v3), first make sure you have access to the etcd cluster and a backup before making changes.
Kubernetes stores Services under /registry/services/ and EndpointSlices under /registry/endpointslices/. You can list all keys with:
To inspect a specific resource, use:
For example, to see all EndpointSlices for a namespace (e.g., default):
Look for multiple keys with similar names or the same service name/namespace. To check if multiple Service objects exist for the same logical service, compare the key suffixes and inspect their values for metadata like
metadata.nameandmetadata.namespace.To match EndpointSlices to Services, inspect the EndpointSlice value and look for labels like
kubernetes.io/service-nameor ownerReferences. If you find more than one Service object referencing the same EndpointSlice, or multiple EndpointSlices with the same service label, that's a sign of duplication or stale resources.If you confirm a duplicate or orphaned entry, you can remove it with:
Be careful—deleting the wrong key can break your cluster. Always back up etcd first.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
fuad00 commentedon Nov 4, 2025
i think i'm getting same issue. Any improvement?
kinseii commentedon Nov 6, 2025
I got around the problem using
exposeType: Proxied. I also simply recreated the cluster differently, with a new name. I haven't had time yet to figure out how to connect to the root etcd. However, creating a separate etcd for each tenant might also be a good idea.kinseii commentedon Nov 12, 2025
I think I understand what the problem was. We use external-dns to register the ingress URL in DNS. It seems that because txtOwnerID was the same on different tenant clusters, MetalLB couldn't determine the active endpoint.
Alekperov Fuad (@fuad00) If your problem has been resolved, may I close the thread?
fuad00 commentedon Nov 12, 2025
solved!