Skip to content

Gateway API in Kubernetes app fails to provision LoadBalancer #1017

Closed as not planned
Closed as not planned
Gateway API in Kubernetes app fails to provision LoadBalancer#1017
Assignees
kvaps

Description

@zdenekjanda
Zdenek Janda (zdenekjanda)
opened on Jun 2, 2025

We deploy kubernetes app in 0.31.1. Gateway API is sucessfully enabled by install option and works ok.
We deploy test app as in cilium documentation:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml

Then, gateway is provisioned

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.4/examples/kubernetes/gateway/basic-http.yaml

We confirm sucessfull deployment

kubectl get gateway my-gateway
NAME         CLASS    ADDRESS       PROGRAMMED   AGE
my-gateway   cilium   172.27.73.0   True         29m

However, in parent cluster although its provisioned, its not working nor its propagated by metallb. Also, when i try to provision another loadbalancer in workload cluster, it will fail and stuck entirely in pending

kubectl get svc | grep Load
cilium-gateway-my-gateway   LoadBalancer   10.95.233.184   172.27.73.0   80:30869/TCP   32m
hello-world-service         LoadBalancer   10.95.26.5      <pending>     80:30519/TCP   20m

Something is clearly broken with gateway API unable to correctly work with parent cluster. I can provide testing environment with this issue. Should we test enable gateway API in parent cluster, maybe this will resolve issue ?

Activity

  1. Timofei Larkin (lllamnyp)
    assigned
    Andrei Kvapil (kvaps)
    on Jun 2, 2025

  2. kvaps commented on Jun 2, 2025

    @kvaps
    Andrei Kvapil (kvaps)
    on Jun 2, 2025
    Member

    Hey, cloud you please:

    • try recreation of service
    • provide kubectl describe for this service
    • provide kubevirt ccm logs for your cluster

  3. zdenekjanda commented on Jun 5, 2025

    @zdenekjanda
    Zdenek Janda (zdenekjanda)
    on Jun 5, 2025
    ContributorAuthor

    I have enabled gateway api at parent cluster and its working 100% there, but it didnt helped with this issue. Basically the kubevirt ccm is crashing when its about to configure worker cluster gateway api loadbalancer: Here is logs from kubevirt ccm:

    kubectl logs pod/kubernetes-gatewayapi1-kccm-5bf5c9f4f6-6xsc4 -n tenant-root
I0605 16:47:40.459970       1 serving.go:348] Generated self-signed cert in-memory
I0605 16:47:42.947784       1 serving.go:348] Generated self-signed cert in-memory
I0605 16:47:43.925093       1 controllermanager.go:168] Version: v0.0.0-master+$Format:%H$
I0605 16:47:43.940586       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0605 16:47:43.940681       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0605 16:47:43.940758       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0605 16:47:43.940826       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0605 16:47:43.940840       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0605 16:47:43.940925       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0605 16:47:43.942794       1 secure_serving.go:213] Serving securely on [::]:10258
I0605 16:47:43.942915       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0605 16:47:43.943309       1 leaderelection.go:250] attempting to acquire leader lease kube-system/cloud-controller-manager...
I0605 16:47:44.041425       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0605 16:47:44.041550       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0605 16:47:44.041664       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0605 16:48:00.995932       1 leaderelection.go:260] successfully acquired lease kube-system/cloud-controller-manager
I0605 16:48:00.996287       1 event.go:307] "Event occurred" object="kube-system/cloud-controller-manager" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="kubernetes-gatewayapi1-kccm-5bf5c9f4f6-6xsc4_7badc5f5-8d1b-4ab9-9011-d64e7a877328 became leader"
I0605 16:48:01.013650       1 controllermanager.go:337] Started "cloud-node-lifecycle-controller"
I0605 16:48:01.013936       1 node_lifecycle_controller.go:113] Sending events to api server
I0605 16:48:01.014651       1 controllermanager.go:337] Started "service-lb-controller"
W0605 16:48:01.014687       1 core.go:111] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0605 16:48:01.014716       1 controllermanager.go:325] Skipping "node-route-controller"
I0605 16:48:01.014772       1 kubevirteps.go:32] Starting kubevirt_eps_controller.
I0605 16:48:01.014788       1 kubevirteps.go:45] Setting up tenant client.
I0605 16:48:01.014933       1 controller.go:231] Starting service controller
I0605 16:48:01.014989       1 shared_informer.go:311] Waiting for caches to sync for service
I0605 16:48:01.015271       1 kubevirteps.go:54] Setting up infra client.
I0605 16:48:01.016251       1 kubevirteps.go:102] Setting up kubevirtEPSController
I0605 16:48:01.016331       1 kubevirteps.go:117] Initializing kubevirtEPSController
I0605 16:48:01.016497       1 kubevirteps.go:125] Running kubevirtEPSController
W0605 16:48:01.016534       1 controllermanager.go:325] Skipping "kubevirt_eps_controller"
I0605 16:48:01.016654       1 kubevirteps_controller.go:273] Starting kubevirt_eps_controller
I0605 16:48:01.016767       1 shared_informer.go:311] Waiting for caches to sync for kubevirt_eps_controller
I0605 16:48:01.017463       1 controllermanager.go:337] Started "cloud-node-controller"
I0605 16:48:01.017594       1 node_controller.go:165] Sending events to api server.
I0605 16:48:01.017718       1 node_controller.go:174] Waiting for informer caches to sync
I0605 16:48:01.123262       1 shared_informer.go:318] Caches are synced for service
I0605 16:48:01.125129       1 event.go:307] "Event occurred" object="default/cilium-gateway-worker-gateway" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0605 16:48:01.172537       1 event.go:307] "Event occurred" object="default/cilium-gateway-worker-gateway" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
I0605 16:48:01.174158       1 kubevirteps_controller.go:118] Service added: tenant-root/a42d371bde2194e648e5f2e4196363bc
I0605 16:48:01.174428       1 kubevirteps_controller.go:118] Service added: tenant-root/root-ingress-controller
I0605 16:48:01.217026       1 shared_informer.go:318] Caches are synced for kubevirt_eps_controller
I0605 16:48:01.217110       1 kubevirteps_controller.go:420] Reconciling: a42d371bde2194e648e5f2e4196363bc
I0605 16:48:01.217137       1 kubevirteps_controller.go:377] Searching for EndpointSlices in infra cluster for service tenant-root/a42d371bde2194e648e5f2e4196363bc
I0605 16:48:01.224215       1 kubevirteps_controller.go:358] Searching for endpoints on tenant cluster kubernetes-gatewayapi1 for service cilium-gateway-worker-gateway in namespace default.
I0605 16:48:01.229690       1 kubevirteps_controller_utils.go:62] Adding EndpointSlice cilium-gateway-worker-gateway-xjf96 to the tenantEPSTracker
E0605 16:48:01.229978       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 548 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x20226c0?, 0x3952320})
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001681e00?})
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x20226c0, 0x3952320})
	/usr/local/go/src/runtime/panic.go:884 +0x213
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).getDesiredEndpoints(0xc001456000, 0xc0004e8000, {0xc000251a40, 0x1, 0x21df200?})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:795 +0xb84
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).reconcileByAddressType(0xc001456000, 0xc0004e8000, {0xc000251a40, 0x1, 0x1}, {0x0, 0x0, 0x2392c4b?}, {0x2380ce8, 0x4})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:551 +0x253
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).reconcile(0xc001456000, {0x2716db8, 0xc00033e230}, 0x0?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:506 +0xda5
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).processNextItem(0xc001456000, {0x2716db8, 0xc00033e230})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:308 +0x128
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).runWorker(...)
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:296
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000679500?, {0x26f6ea0, 0xc001453e30}, 0x1, 0xc000679500)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00033e230?, 0x3b9aca00, 0x0, 0x0?, 0x918100?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2716db8, 0xc00033e230}, 0xc000730020, 0x8ff4de?, 0xc001778d80?, 0x0?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x2716db8?, 0xc00033e230?}, 0x9179c6?, 0xc0001f1400?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170 +0x2b
created by kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).Run
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:288 +0x514
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d8b584]

goroutine 548 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001681e00?})
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x20226c0, 0x3952320})
	/usr/local/go/src/runtime/panic.go:884 +0x213
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).getDesiredEndpoints(0xc001456000, 0xc0004e8000, {0xc000251a40, 0x1, 0x21df200?})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:795 +0xb84
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).reconcileByAddressType(0xc001456000, 0xc0004e8000, {0xc000251a40, 0x1, 0x1}, {0x0, 0x0, 0x2392c4b?}, {0x2380ce8, 0x4})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:551 +0x253
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).reconcile(0xc001456000, {0x2716db8, 0xc00033e230}, 0x0?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:506 +0xda5
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).processNextItem(0xc001456000, {0x2716db8, 0xc00033e230})
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:308 +0x128
kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).runWorker(...)
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:296
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000679500?, {0x26f6ea0, 0xc001453e30}, 0x1, 0xc000679500)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00033e230?, 0x3b9aca00, 0x0, 0x0?, 0x918100?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2716db8, 0xc00033e230}, 0xc000730020, 0x8ff4de?, 0xc001778d80?, 0x0?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:259 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x2716db8?, 0xc00033e230?}, 0x9179c6?, 0xc0001f1400?)
	/go/src/kubevirt.io/cloud-provider-kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:170 +0x2b
created by kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps.(*Controller).Run
	/go/src/kubevirt.io/cloud-provider-kubevirt/pkg/controller/kubevirteps/kubevirteps_controller.go:288 +0x514

    Also, endpoint gets like halfway provisioned, but the ip will never propagate to worker cluster:

    kubectl describe svc a42d371bde2194e648e5f2e4196363bc -n tenant-root
Name:                     a42d371bde2194e648e5f2e4196363bc
Namespace:                tenant-root
Labels:                   cluster.x-k8s.io/cluster-name=kubernetes-gatewayapi1
                          cluster.x-k8s.io/tenant-service-name=cilium-gateway-worker-gateway
                          cluster.x-k8s.io/tenant-service-namespace=default
Annotations:              metallb.io/ip-allocated-from-pool: private-pool
Selector:                 <none>
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.108.218
IPs:                      10.96.108.218
LoadBalancer Ingress:     172.27.73.1 (VIP)
Port:                     port-80  80/TCP
TargetPort:               32676/TCP
NodePort:                 port-80  30115/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason       Age   From                Message
  ----    ------       ----  ----                -------
  Normal  IPAllocated  14m   metallb-controller  Assigned IP ["172.27.73.1"]

    Any suggestions on how to proceed ? Fresh provisioned system is available for debug

  4. kvaps commented on Jun 11, 2025

    @kvaps
    Andrei Kvapil (kvaps)
    on Jun 11, 2025
    Member

    Hey Zdenek, sorry for being late with my answer, it seems you have a panic in kccm.
    The upstream project it https://github.com/kubevirt/cloud-provider-kubevirt/

    I’ll try to fix the panic and prepare the PR for you.

  5. kvaps commented on Jun 11, 2025

    @kvaps
    Andrei Kvapil (kvaps)
    on Jun 11, 2025
    Member

    I just prepared fix for upstream

  6. dosubot commented on Oct 20, 2025

    @dosubot
    dosubotbot
    on Oct 20, 2025 – with DosuBot

    Hi, Zdenek Janda (@zdenekjanda). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

    Issue Summary

    • You reported LoadBalancer provisioning issues and kubevirt cloud-controller-manager crashing with a nil pointer panic when enabling Gateway API in a Kubernetes app using Cilium 0.31.1.
    • The problem persisted even after enabling Gateway API in the parent cluster.
    • Detailed logs showed the panic originated in kubevirt's kubevirteps controller.
    • The maintainer identified this as a bug in the upstream kubevirt/cloud-provider-kubevirt project.
    • Fixes have been prepared and linked as pull requests for both the upstream and cozystack repositories.

    Next Steps

    • Please confirm if this issue is still relevant with the latest version of the cozystack repository by commenting here.
    • If no further updates are provided, I will automatically close this issue in 7 days.

    Thank you for your understanding and contribution!

  7. dosubot
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale
    on Oct 20, 2025
  8. dosubot
    closed this as not plannedon Oct 27, 2025
  9. dosubot
    removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale
    on Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions