Skip to content

[occm] openstack.(*LbaasV2).ensureAndUpdateOctaviaSecurityGroup encounters SIGSEGV #3101

Closed
#3163
gophercloud/gophercloud
#3897
Closed
[occm] openstack.(*LbaasV2).ensureAndUpdateOctaviaSecurityGroup encounters SIGSEGV#3101
#3163
gophercloud/gophercloud
#3897
Labels
kind/bugCategorizes issue or PR as related to a bug.

Description

@judge-red
judge-red
opened on Apr 29, 2026

/kind bug

What happened:

We have many Kubernetes clusters running, which are configured more or less identical as far as the occm is concerned. But the occm for one is in a CrashLoopBackOff. Presumably something that was done with OpenStack-powered Kubernetes resource in the cluster or that was done on the OpenStack side triggers this, but I have no idea where to even start looking. Thus I can only provide the most recent log with the SIGSEGV we're envountering. Please advise what additional information we should provide.

occm log:

Details 
Defaulted container "cloud-controller-manager" out of: cloud-controller-manager, copy-http-prober (init)
{"level":"info","time":"2026-04-29T12:55:45.276Z","logger":"http-prober","caller":"http-prober/main.go:137","msg":"Probing","attempt":1,"max-attempts":100,"target":"https://apiserver-externa
l.cluster-znqzsbl25p.svc.cluster.local./healthz"}
{"level":"info","time":"2026-04-29T12:55:45.283Z","logger":"http-prober","caller":"http-prober/main.go:126","msg":"Hostname resolved","hostname":"apiserver-external.cluster-znqzsbl25p.svc.cl
uster.local.","address":"10.111.75.178:443"}
{"level":"info","time":"2026-04-29T12:55:45.301Z","logger":"http-prober","caller":"http-prober/main.go:150","msg":"Endpoint is available"}
I0429 12:55:45.624917       1 serving.go:386] Generated self-signed cert in-memory
I0429 12:55:46.043872       1 serving.go:386] Generated self-signed cert in-memory
W0429 12:55:46.398830       1 authentication.go:476] failed to read in-cluster kubeconfig for delegated authentication: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file
 or directory
W0429 12:55:46.398854       1 authentication.go:368] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, s
o client certificate authentication won't work.
W0429 12:55:46.398861       1 authentication.go:392] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in
kube-system, so request-header client certificate authentication won't work.
W0429 12:55:46.398874       1 authorization.go:225] failed to read in-cluster kubeconfig for delegated authorization: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file o
r directory
W0429 12:55:46.398885       1 authorization.go:193] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0429 12:55:46.782755       1 controllermanager.go:160] Version: v1.35.0
I0429 12:55:46.786969       1 secure_serving.go:211] Serving securely on [::]:10258
I0429 12:55:46.787067       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I0429 12:55:46.787214       1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="kube-system/cloud-controller-manager"
I0429 12:56:03.197339       1 leaderelection.go:272] "Successfully acquired lease" lock="kube-system/cloud-controller-manager"
I0429 12:56:03.197819       1 event.go:389] "Event occurred" object="kube-system/cloud-controller-manager" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason=
"LeaderElection" message="openstack-cloud-controller-manager-5658d675f4-99n6t_634ec517-8a0c-4aeb-8e96-b2132274a8b9 became leader"
I0429 12:56:03.201597       1 openstack.go:424] Setting up informers for Cloud
I0429 12:56:03.201839       1 controllermanager.go:310] Starting "cloud-node-lifecycle-controller"
I0429 12:56:03.249162       1 controllermanager.go:329] Started "cloud-node-lifecycle-controller"
I0429 12:56:03.249184       1 controllermanager.go:310] Starting "service-lb-controller"                                                                                                      I0429 12:56:03.249601       1 node_lifecycle_controller.go:112] Sending events to api server
W0429 12:56:03.316564       1 openstack.go:362] Failed to create an OpenStack Secret client: unable to initialize keymanager client for region zhw: No suitable endpoint could be found in the
 service catalog.
I0429 12:56:03.316612       1 openstack.go:373] Claiming to support LoadBalancer
I0429 12:56:03.316662       1 controllermanager.go:329] Started "service-lb-controller"
I0429 12:56:03.316695       1 controllermanager.go:310] Starting "node-route-controller"
I0429 12:56:03.317895       1 controller.go:235] Starting service controller
I0429 12:56:03.317993       1 shared_informer.go:370] "Waiting for caches to sync"
W0429 12:56:03.376811       1 openstack.go:408] Error initialising Routes support: router-id not set in cloud provider config
W0429 12:56:03.376834       1 core.go:111] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0429 12:56:03.376842       1 controllermanager.go:317] Skipping "node-route-controller"
I0429 12:56:03.376848       1 controllermanager.go:310] Starting "cloud-node-controller"
I0429 12:56:03.388974       1 controllermanager.go:329] Started "cloud-node-controller"
I0429 12:56:03.389240       1 node_controller.go:176] Sending events to api server.
I0429 12:56:03.389312       1 node_controller.go:185] Waiting for informer caches to sync
I0429 12:56:03.418597       1 shared_informer.go:377] "Caches are synced"
I0429 12:56:03.418778       1 loadbalancer.go:1863] "EnsureLoadBalancer" cluster="znqzsbl25p" service="cilium-gateway/cilium-gateway-gateway-auth-dmz-v4"
I0429 12:56:03.418980       1 event.go:389] "Event occurred" object="cilium-gateway/cilium-gateway-gateway-auth-dmz-v4" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="Ensu
ringLoadBalancer" message="Ensuring load balancer"
E0429 12:56:04.107997       1 panic.go:262] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer
dereference\"" stacktrace=<
        goroutine 247 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2d44480, 0xc0009ab680}, {0x24b2940, 0x43db840})
                k8s.io/[email protected]/pkg/util/runtime/runtime.go:132 +0xbc
        k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2d444b8, 0xc000370690}, {0x24b2940, 0x43db840}, {0x0, 0x0, 0xc000806cd0?})
                k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0x116
        k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x2d444b8, 0xc000370690}, {0x0, 0x0, 0x0})
                k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
        panic({0x24b2940?, 0x43db840?})
                runtime/panic.go:783 +0x132
        k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).ensureAndUpdateOctaviaSecurityGroup(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3
e0, 0x2, 0x2}, 0xc000aaba20)
                k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer_sg.go:248 +0x73c
        k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).ensureOctaviaLoadBalancer(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3e0, 0x2, 0
x2})
                k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer.go:1845 +0x1fc5
        k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).EnsureLoadBalancer(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3e0, 0x2, 0x2})
                k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer.go:1864 +0x2c5
        k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408)
                k8s.io/[email protected]/controllers/service/controller.go:453 +0x132
        k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408, {0xc000a2fd40, 0x31})
                k8s.io/[email protected]/controllers/service/controller.go:409 +0x745
        k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408, {0xc000a2fd40, 0x31})
                k8s.io/[email protected]/controllers/service/controller.go:340 +0x13c
        k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc000942a90, {0x2d444b8, 0xc000370690}, {0xc000a2fd40, 0x31})
                k8s.io/[email protected]/controllers/service/controller.go:897 +0x1ac
        k8s.io/cloud-provider/controllers/service.(*Controller).processNextServiceItem(0xc000942a90, {0x2d444b8, 0xc000370690})
                k8s.io/[email protected]/controllers/service/controller.go:291 +0xed
        k8s.io/cloud-provider/controllers/service.(*Controller).serviceWorker(...)
                k8s.io/[email protected]/controllers/service/controller.go:258
        k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x2d444b8?, 0xc000370690?}, 0xc000a99f50?)
                k8s.io/[email protected]/pkg/util/wait/backoff.go:255 +0x51
        k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x2d444b8, 0xc000370690}, 0xc0009db330, {0x2d1d460, 0xc000a99f50}, 0x1)
                k8s.io/[email protected]/pkg/util/wait/backoff.go:256 +0xe5
        k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2d444b8, 0xc000370690}, 0xc0009db330, 0x3b9aca00, 0x0, 0x1)
                k8s.io/[email protected]/pkg/util/wait/backoff.go:223 +0x8f
        k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
                k8s.io/[email protected]/pkg/util/wait/backoff.go:172
        created by k8s.io/cloud-provider/controllers/service.(*Controller).Run in goroutine 213
                k8s.io/[email protected]/controllers/service/controller.go:245 +0x474
 >
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x213809c]

goroutine 247 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2d444b8, 0xc000370690}, {0x24b2940, 0x43db840}, {0x0, 0x0, 0xc000806cd0?})
        k8s.io/[email protected]/pkg/util/runtime/runtime.go:114 +0x1a9
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x2d444b8, 0xc000370690}, {0x0, 0x0, 0x0})
        k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x24b2940?, 0x43db840?})
        runtime/panic.go:783 +0x132
k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).ensureAndUpdateOctaviaSecurityGroup(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3e0, 0x2, 0x2}, 0xc000aaba20)
        k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer_sg.go:248 +0x73c
k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).ensureOctaviaLoadBalancer(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3e0, 0x2, 0x2})
        k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer.go:1845 +0x1fc5
k8s.io/cloud-provider-openstack/pkg/openstack.(*LbaasV2).EnsureLoadBalancer(0xc0000e7520, {0x2d444b8, 0xc000370690}, {0x7ffff49e81a8, 0xa}, 0xc000225408, {0xc0009db3e0, 0x2, 0x2})
        k8s.io/cloud-provider-openstack/pkg/openstack/loadbalancer.go:1864 +0x2c5
k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408)
        k8s.io/[email protected]/controllers/service/controller.go:453 +0x132
k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408, {0xc000a2fd40, 0x31})
        k8s.io/[email protected]/controllers/service/controller.go:409 +0x745
k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc000942a90, {0x2d444b8, 0xc000370690}, 0xc000225408, {0xc000a2fd40, 0x31})
        k8s.io/[email protected]/controllers/service/controller.go:340 +0x13c
k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc000942a90, {0x2d444b8, 0xc000370690}, {0xc000a2fd40, 0x31})
        k8s.io/[email protected]/controllers/service/controller.go:897 +0x1ac
k8s.io/cloud-provider/controllers/service.(*Controller).processNextServiceItem(0xc000942a90, {0x2d444b8, 0xc000370690})
        k8s.io/[email protected]/controllers/service/controller.go:291 +0xed
k8s.io/cloud-provider/controllers/service.(*Controller).serviceWorker(...)
        k8s.io/[email protected]/controllers/service/controller.go:258
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x2d444b8?, 0xc000370690?}, 0xc000a99f50?)
        k8s.io/[email protected]/pkg/util/wait/backoff.go:255 +0x51
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x2d444b8, 0xc000370690}, 0xc0009db330, {0x2d1d460, 0xc000a99f50}, 0x1)
        k8s.io/[email protected]/pkg/util/wait/backoff.go:256 +0xe5
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x2d444b8, 0xc000370690}, 0xc0009db330, 0x3b9aca00, 0x0, 0x1)
        k8s.io/[email protected]/pkg/util/wait/backoff.go:223 +0x8f
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
        k8s.io/[email protected]/pkg/util/wait/backoff.go:172
created by k8s.io/cloud-provider/controllers/service.(*Controller).Run in goroutine 213
        k8s.io/[email protected]/controllers/service/controller.go:245 +0x474

What you expected to happen:

No SIGSEGV :)

How to reproduce it:

Unknown

Anything else we need to know?:

cloud-config:

Details 
[Global]
auth-url = "https://identity.openstack.url/v3"
region = "foo"
application-credential-id = "hello"
application-credential-secret = "world"

[LoadBalancer]
manage-security-groups = true
lb-version = "v2"
lb-provider = "ovn"
subnet-id = ""
floating-network-id = "xyz"
lb-method = "SOURCE_IP_PORT"

[BlockStorage]
ignore-volume-az = false
trust-device-path = false
bs-version = "auto"

Environment:

  • openstack-cloud-controller-manager(or other related binary) version: registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.35.0
  • OpenStack version: Caracal 2024.1
  • Others:

Activity

  1. k8s-ci-robot
    added
    kind/bugCategorizes issue or PR as related to a bug.
    on Apr 29, 2026

  2. kayrus commented on Apr 29, 2026

    @kayrus
    kayrus
    on Apr 29, 2026
    Contributor

    I reproduced this issue with gophercloud: when the subnet ID is set to an empty string, the API call ends up at /v2.0/subnets/ (list endpoint) instead of /v2.0/subnets/<id> (get endpoint). The list response returns {"subnets":[...]} which doesn't match the expected single-resource shape {"subnet":...}, causing the returned subnet resource to be nil.

    This suggests svcConf.lbMemberSubnetID is empty in your case, worth double-checking how it's being populated.

  3. judge-red commented on Apr 29, 2026

    @judge-red
    judge-red
    on Apr 29, 2026
    Author

    In the meantime, we have a second cluster with the same issue. And both use the same setup - which is a bit more restricted, on an OpenStack level, than all other clusters, i.e. the security of their projects is increased.

    Anyway, we noticed that both set the loadbalancer.openstack.org/network-id annotation on their Services, but not the loadbalancer.openstack.org/subnet-id annotation. Adding the latter helped with one of the clusters (not yet tested in the other).

    Usually we set neither of these annotations, but these clusters have a special networking setup, as per what I mentioned initially. Not sure, is it expected that both must be set? The openstack CLI is happy with just either (or a port ID). I wonder, does the subnet looking for the network fail? Maybe because the network is shared from a separate project or maybe an RBAC problem? I tried running into a wall with the openstack CLI using the same credentials, but I seem to being able to query everything ... but not sure what exactly the occm queries, is there anything I could try to reproduce it with the CLI?

  4. SAY-5
    added a commit that references this issue on May 5, 2026

    loadbalancer_sg: nil-check subnet.Extract result

    c52bd29

  5. 234u34k commented on Jul 29, 2026

    @234u34k
    234u34k
    on Jul 29, 2026
    Contributor

    I traced this through the existing-load-balancer path. The review on #3105 established that Gophercloud should not return a nil resource without an error, so I would address that upstream rather than repeat the downstream nil check here.

    There is also a separate OCCM issue before that call: with a network-only Service, existing-LB reconciliation can reach managed security-group handling with no member subnet ID. I propose resolving a subnet only for that security-group operation, using the existing member subnet when set, otherwise the resolved load balancer's VipSubnetID, and returning an error before any Neutron request if neither is available.

    I would keep this value out of svcConf.lbMemberSubnetID so member requests continue to omit subnet_id where supported. I can send the Gophercloud guard and tests separately, then a focused OCCM regression fix covering both Ensure and Update. Does anyone object to that split?

  6. kayrus commented on Jul 29, 2026

    @kayrus
    kayrus
    on Jul 29, 2026
    Contributor

    @234u34k Thanks for the reminder. I've submitted a fix to gophercloud: gophercloud/gophercloud#3897

  7. 234u34k commented on Jul 29, 2026

    @234u34k
    234u34k
    on Jul 29, 2026
    Contributor

    Thanks. Once consumed here, that covers the nil result and turns the panic into an extraction error. It still leaves the network-only path failing because OCCM reaches subnets.Get with an empty member subnet ID. Do you consider #3897 sufficient for #3101, or should I proceed with the scoped OCCM fallback described above?

  8. kayrus commented on Jul 29, 2026

    @kayrus
    kayrus
    on Jul 29, 2026
    Contributor

    It still leaves the network-only path failing because OCCM reaches subnets.Get with an empty member subnet ID.

    gophercloud/gophercloud#3897 covers every API call with an empty ID

  9. judge-red commented on Jul 30, 2026

    @judge-red
    judge-red
    on Jul 30, 2026
    Author

    Thanks for tackling this issue, folks! I'll admit I forgot about it as nobody has done the same ever since, but I'm sure it's just a matter of time as network setups of some of our team are getting ever more complex for security / compliance reasons, so this is still much appreciated.

  10. kubernetes-prow
    closed this as completedin #3163on Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions