Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster

Describe the bug
Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster:
Example:

---
apiVersion: v1
kind: Service
metadata:
  name: lb-test1
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: app-test1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-test1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-test1
  template:
    metadata:
      labels:
        app: app-test1
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80

These IPs have already been used in previous versions of CozyStack. If I create multiple LBs, and the pool will issue IPs that have not been used before, then everything is fine.

Debugging shows that metallb complains that there are no Endpoints, although there are EPs inside the tenant cluster, and if I do port forwarding to svc LB inside tenant cluster, everything works:

$ kc get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
lb-test1     LoadBalancer   10.95.164.25    2xx.xxx.xxx.xxx    80:32479/TCP   42s
$ kc port-forward svc/lb-test1 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080

IPs that are successfully announced have an endpoint. This endpoint is the IP address of virt-launcher pod in the root cluster, i.e., the node of the tenant k8s. And the NodePort is the external port of that root k8s node where virt-launcher is located. This NodePort is open externally if you try to connect to it on the IP node of root k8s.
Also, metallb-controller and metalb-speaker respond, and there are entries in Events.

# kc describe svc a375e7eaa3e994966b8920136c013b45 -n tenant-qwerty
Name:                     a375e7eaa3e994966b8920136c013b45
=== cut ===
Type:                     LoadBalancer
TargetPort:               30815/TCP
NodePort:                 <unset>  32611/TCP
Endpoints:                10.244.3.155:30815
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason        Age    From                Message
  ----    ------        ----   ----                -------
  Normal  IPAllocated   9m38s  metallb-controller  Assigned IP ["2xx.xxx.xxx.xxx"]
  Normal  nodeAssigned  9m38s  metallb-speaker     announcing from node "qwerty-xxx-xxx-xxx-1" with protocol "layer2"

# nc -v 2xx.xxx.xxx.xxx 32611
(UNKNOWN) [2xx.xxx.xxx.xxx] 32611 (?) open

The external node port of the problematic svc is unavailable, and the speakers have no records:

# kc describe svc -n tenant-qwerty a5ed58225d6b941e3885fe37d59801d2
Name:                     a5ed58225d6b941e3885fe37d59801d2
=== cut ===
Type:                     LoadBalancer
Port:                     <unset>  80/TCP
TargetPort:               32479/TCP
NodePort:                 <unset>  31985/TCP
Endpoints:                <none>
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
  Normal  IPAllocated  46m   metallb-controller  Assigned IP ["2xx.xxx.xxx.yyy"]

# nc -v 2xx.xxx.xxx.zzz 31985
(UNKNOWN) [2xx.xxx.xxx.zzz] 31985 (?) : Connection refused

And in the logs, there is a message saying that there are no endpoints.

{"caller":"layer2_controller.go:84","event":"shouldannounce","ips":["2xx.xxx.xxx.yyy"],"level":"debug","message":"failed no active endpoints","pool":"cozystack","protocol":"l2","service":"tenant-qwerty/a5ed58225d6b941e3885fe37d59801d2","ts":"2025-10-31T14:45:43Z"}

Please tell me what else I can check?

Environment

Cozystack version: 0.37.2
Provider: on-prem

This issue is likely related to how endpoints are propagated from tenant clusters to the root cluster, especially for IPs that were previously used in older CozyStack versions. There's a known bug where a panic in the kubevirt cloud controller manager prevents endpoints from being correctly created in the root cluster, which causes MetalLB to report "no active endpoints" and not announce the IP externally—even though everything works inside the tenant cluster. This matches your symptoms and has been discussed in detail here.

To troubleshoot, check for errors or panics in the kubevirt cloud controller manager logs in the root cluster. If you see signs of a crash or nil pointer dereference, this is almost certainly the cause. The fix for this was prepared upstream and merged into CozyStack (PR link), so make sure your environment includes this patch.

If you suspect stale resources, you can manually check for leftover endpoints, NodePorts, or IP allocations in both clusters. Use kubectl get endpoints -A and kubectl get svc -A in both root and tenant clusters to look for services with missing endpoints or NodePorts. If you find any, try deleting and recreating the affected LoadBalancer service to force a fresh allocation and endpoint mapping.

Also, confirm that your MetalLB IP pool only contains unused IPs and that routing is correct for all IPs in the pool (docs). Sometimes, previously used IPs may be blocked by lingering resources or misconfigured routes.

If the issue persists after cleanup and patching, review the events and logs for MetalLB and kubevirt controllers for clues about resource propagation failures.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

Thanks to the bot, I checked the KCCM log and found a multiple svc error. However, I don't have multiple svc, so these are probably old entries in etcd.

# kc logs kubernetes-qwerty-xxx-xxx-xxx-1-kccm-644dcdbdc5-p9gfm -n tenant-qwerty
I1031 16:09:10.967274       1 event.go:307] "Event occurred" object="default/lb-test1" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I1031 16:09:10.986155       1 kubevirteps_controller.go:117] Service added: tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:10.986209       1 kubevirteps_controller.go:419] Reconciling: ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:10.986234       1 kubevirteps_controller.go:376] Searching for EndpointSlices in infra cluster for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:10.988637       1 kubevirteps_controller.go:357] Searching for endpoints on tenant cluster kubernetes-qwerty-xxx-xxx-xxx-1 for service lb-test1 in namespace default.
I1031 16:09:10.990292       1 kubevirteps_controller.go:124] Service updated: tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:10.992723       1 kubevirteps_controller_utils.go:62] Adding EndpointSlice lb-test1-nxzlk to the tenantEPSTracker
I1031 16:09:10.992758       1 kubevirteps_controller.go:802] Desired nodes for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927: []
I1031 16:09:10.992794       1 kubevirteps_controller.go:419] Reconciling: ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:10.992817       1 kubevirteps_controller.go:376] Searching for EndpointSlices in infra cluster for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:11.008071       1 kubevirteps_controller.go:124] Service updated: tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:11.089502       1 kubevirteps_controller.go:357] Searching for endpoints on tenant cluster kubernetes-qwerty-xxx-xxx-xxx-1 for service lb-test1 in namespace default.
I1031 16:09:11.095813       1 kubevirteps_controller_utils.go:62] Adding EndpointSlice lb-test1-nxzlk to the tenantEPSTracker
I1031 16:09:11.095850       1 kubevirteps_controller.go:802] Desired nodes for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927: []
I1031 16:09:11.095888       1 kubevirteps_controller.go:419] Reconciling: ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:11.095912       1 kubevirteps_controller.go:376] Searching for EndpointSlices in infra cluster for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927
I1031 16:09:11.098119       1 kubevirteps_controller.go:357] Searching for endpoints on tenant cluster kubernetes-qwerty-xxx-xxx-xxx-1 for service lb-test1 in namespace default.
I1031 16:09:11.102150       1 kubevirteps_controller_utils.go:62] Adding EndpointSlice lb-test1-nxzlk to the tenantEPSTracker
I1031 16:09:11.102182       1 kubevirteps_controller.go:802] Desired nodes for service tenant-qwerty/ab0bfc6fcc5734eeca75375a0a996927: []
I1031 16:09:13.165503       1 kubevirteps_controller.go:159] get Infra Service for Tenant EndpointSlice: default/lb-test1-nxzlk
E1031 16:09:13.168818       1 kubevirteps_controller.go:342] Multiple services found for tenant endpoint slice lb-test1-nxzlk in namespace default
E1031 16:09:13.168848       1 kubevirteps_controller.go:163] Failed to get Service in Infra cluster for EndpointSlice default/lb-test1-nxzlk: multiple services found for tenant endpoint slice
I1031 16:09:15.988253       1 event.go:307] "Event occurred" object="default/lb-test1" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"

The "multiple services found for tenant endpoint slice" error in the CCM logs happens when the controller detects more than one Service resource associated with the same EndpointSlice in the infra cluster. This usually means there are stale or orphaned Service entries lingering in etcd or the infra cluster—often from previous CozyStack versions or failed deletions—even if they're not visible with standard kubectl commands. These hidden resources can block correct endpoint mapping and break MetalLB LoadBalancer provisioning, especially for IPs reused from older clusters or services.

There isn't a built-in automated cleanup for this in CozyStack. The recommended approach is to manually identify and remove any leftover Service or EndpointSlice resources in the infra cluster's etcd. You can try:

Listing all Services and EndpointSlices in the affected namespace with kubectl get svc -A and kubectl get endpointslices -A in both tenant and infra clusters.
If you suspect hidden resources, you may need to use etcdctl directly on the infra cluster's etcd to search for and remove orphaned Service or EndpointSlice keys.
Deleting and recreating the affected LoadBalancer Service in the tenant cluster can sometimes force a fresh mapping and clear the error.

This issue is more likely to occur with IPs that were previously allocated, so cleaning up these stale resources should resolve the mapping conflict and restore MetalLB functionality. For more background, see the discussion and upstream fix here: cozystack/cozystack#1017.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) How can I determine that endpointslice has multiple svc?

The CCM error "multiple services found for tenant endpoint slice" means the controller found more than one Service resource associated with a single EndpointSlice—usually due to stale or orphaned resources in etcd that aren't visible with normal kubectl commands.

To check for this:

List all EndpointSlices and Services in the affected namespace with:

kubectl get endpointslices -n <namespace>
kubectl get svc -n <namespace>

For a specific EndpointSlice (e.g., lb-test1-nxzlk), inspect its labels and ownerReferences:
```
kubectl get endpointslice lb-test1-nxzlk -n <namespace> -o yaml
```
Look for labels like kubernetes.io/service-name or ownerReferences pointing to a Service.
List all Services and check if more than one has a matching label or selector for that EndpointSlice. For example:
```
kubectl get svc -n <namespace> -o yaml | grep lb-test1
```
If you see more than one Service referencing the same EndpointSlice, that's the conflict.

If you suspect hidden resources, you may need to use etcdctl on the infra cluster to search for duplicate Service or EndpointSlice keys. The CCM log itself is a strong indicator—if you see this error, there are definitely multiple Service objects (even if not visible via kubectl) associated with the EndpointSlice in etcd or the infra cluster. Cleaning up these stale resources should resolve the issue.

For more context, see the discussion in cozystack/cozystack#1017.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) how to find duplicates through etcdctl

To find duplicate or orphaned Service and EndpointSlice resources in etcd using etcdctl (v3), first make sure you have access to the etcd cluster and a backup before making changes.

Kubernetes stores Services under /registry/services/ and EndpointSlices under /registry/endpointslices/. You can list all keys with:

etcdctl get /registry/services/ --prefix --keys-only
etcdctl get /registry/endpointslices/ --prefix --keys-only

To inspect a specific resource, use:

etcdctl get <key>

For example, to see all EndpointSlices for a namespace (e.g., default):

etcdctl get /registry/endpointslices/default/ --prefix --keys-only

Look for multiple keys with similar names or the same service name/namespace. To check if multiple Service objects exist for the same logical service, compare the key suffixes and inspect their values for metadata like metadata.name and metadata.namespace.

To match EndpointSlices to Services, inspect the EndpointSlice value and look for labels like kubernetes.io/service-name or ownerReferences. If you find more than one Service object referencing the same EndpointSlice, or multiple EndpointSlices with the same service label, that's a sign of duplication or stale resources.

If you confirm a duplicate or orphaned entry, you can remove it with:

etcdctl del <key>

Be careful—deleting the wrong key can break your cluster. Always back up etcd first.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

i think i'm getting same issue. Any improvement?

I got around the problem using exposeType: Proxied. I also simply recreated the cluster differently, with a new name. I haven't had time yet to figure out how to connect to the root etcd. However, creating a separate etcd for each tenant might also be a good idea.

Alekperov Fuad (@fuad00)

I think I understand what the problem was. We use external-dns to register the ingress URL in DNS. It seems that because txtOwnerID was the same on different tenant clusters, MetalLB couldn't determine the active endpoint.

Alekperov Fuad (@fuad00) If your problem has been resolved, may I close the thread?

solved!

Navigation Menu

Repository navigation

Some IPs in the pool for MetalLB are not available when creating svc `LoadBalancer` in a tenant cluster #1563

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Some IPs in the pool for MetalLB are not available when creating svc LoadBalancer in a tenant cluster #1563

Description

Activity

dosubot commented on Oct 31, 2025

kinseii commented on Oct 31, 2025

dosubot commented on Oct 31, 2025

kinseii commented on Oct 31, 2025

dosubot commented on Oct 31, 2025

kinseii commented on Oct 31, 2025

dosubot commented on Oct 31, 2025

fuad00 commented on Nov 4, 2025

kinseii commented on Nov 6, 2025

kinseii commented on Nov 12, 2025

fuad00 commented on Nov 12, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Some IPs in the pool for MetalLB are not available when creating svc `LoadBalancer` in a tenant cluster #1563