Unable to connect to external nats (gateway/storage-users)

Describe the bug

I'm trying to get up and running with opencloud using the reference deployment architecture from here

I've set the nats server directly from the opencloud binary (opencloud nats server command) and the next step was to get the storage-users service up and running. I can't seem to make it try to connect to an external nats seerver.

With trace logging on the storage-users, I can see this

{"level":"info","service":"storage-users","time":"2026-09-30T09:15:43Z","line":"github.com/opencloud-eu/opencloud/pkg/registry/register.go:19","message":"registering external service eu.opencloud.api.storage-users-d034bb9f-e304-44ba-ab58-fa1caa2c8a2b@10.1.1.82:9157"}
{"level":"fatal","service":"storage-users","error":"Failed to connect to NATS Server: nats: no servers available for connection","time":"2026-09-30T09:15:43Z","line":"github.com/opencloud-eu/opencloud/pkg/registry/register.go:23","message":"Registration error for external service eu.opencloud.api.storage-users"}

But no matter how I configure the endpoint (fqdn, env variables, file configuration) the storage-users doesn't even try to make the connection. I have cilium/hubble to monitor the network traffic and the specific storage-users deployment isn't attempting to connect to the nats server I point it to.
I did try to create a deployment of opencloud with the nats server removed and OC_EVENTS_ENDPOINT configured to my NATS server, and that was able to establish the connection.
Similarly, a busybox container within the same namespace with nc/telnet is able to connect to the external nats just fine.

Steps to reproduce

Have external NATS service running
Start storage-users with the OC_EVENTS_ENDPOINT pointed to the external NATS

Expected behavior

Externalization of any service to function, especially if it's part of the reference architecture
Better logs around connections being established, currently I can't say for sure that the storage-users service is picking up my configuration correctly

Actual behavior

storage-users unable to connect to external nats

Setup

I'm currently in the process of setting up the entire stack... I just have nats, openldap and the storage-users services that I'm trying to spin-up

Details

For the nats service I have the following configuration

loglevel: info
debug:
  addr: 0.0.0.0:9234
  token: ""
  pprof: false
  zpages: false
nats:
  host: 0.0.0.0
  port: 9233
  clusterid: opencloud-cluster
  store_dir: /var/lib/opencloud/nats
  enable_tls: false

I am also passing these environment variables although the documentation says nothing about them, but I thought I would give it a try regardless

OC_SERVICE_ACCOUNT_ID
OC_SERVICE_ACCOUNT_SECRET

The storage-users is configured as follows
THe data_server_url, data_gateway_url and gateway_addr I left as is as I'm still progressing with trying to roll this out. For now I just wanted to see this service come up and become healthy.

loglevel: trace
data_server_url: http://localhost:9158/data
data_gateway_url: http://localhost:9140/data
gateway_addr: 127.0.0.1:9142
debug:
  addr: 0.0.0.0:9159
  token: ""
  pprof: false
  zpages: false
grpc:
  addr: 0.0.0.0:9157
  tls: null
  protocol: tcp
http:
  addr: 0.0.0.0:9158
  protocol: tcp
  prefix: data
  cors:
    ...
skip_user_groups_in_token: false
graceful_shutdown_timeout: 30
driver: posix
drivers:
  posix:
    root: /var/lib/opencloud/storage/users
    personalspacealias_template: {{`'{{.SpaceType}}/{{.User.Username | lower}}'`}}
    personalspacepath_template: {{`users/{{.User.Id.OpaqueId}}`}}
    generalspacealias_template: {{`'{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'`}}
    generalspacepath_template: {{`projects/{{.SpaceId}}`}}
    permissions_endpoint: eu.opencloud.api.settings
    async_uploads: true
    max_quota: 0
    max_acquire_lock_cycles: 0
    lock_cycle_duration_factor: 0
    max_concurrency: 0
transfer_expires: 86400
events:
  endpoint: {{ include "opencloud.fullname" . }}-nats.{{ .Release.Namespace }}.svc.cluster.local:9233
  cluster: opencloud-cluster
  enable_tls: false
  tls_insecure: true
  num_consumers: 0
filemetadata_cache:
  store: memory
  nodes:
  - {{ include "opencloud.fullname" . }}-nats.{{ .Release.Namespace }}.svc.cluster.local:9233
  database: storage-users
  disable_persistence: false
id_cache:
  store: nats-js-kv
  nodes:
  - {{ include "opencloud.fullname" . }}-nats.{{ .Release.Namespace }}.svc.cluster.local:9233
  database: ids-storage-users
  disable_persistence: false
expose_data_server: false
readonly: false
upload_expiration: 86400
max_attempts_rename_file: 0

Above is a Helm template but not a lot of it is parametrized, the service pointing to NATS is set as {{ include "opencloud.fullname" . }}-nats

I did also try changing the id_cache to memory as I wasn't sure which NATS server connection was failing, but that changed nothing.

I also set these env variables (These are generated as part of the install and shared across all components - service account id and secret would be the same value across nats/storage-users, don't think it matters here as nats doesn't seem to need it but thought to point that out)

OC_JWT_SECRET
OC_SERVICE_ACCOUNT_ID
OC_SERVICE_ACCOUNT_SECRET
OC_MACHINE_AUTH_API_KEY
STORAGE_USERS_MOUNT_ID

Additionally I also tried setting this up just in case the file configuration is having issues

OC_EVENTS_ENDPOINT

Hubble just shows a different opencloud-server deployment connecting to NATS. The opencloud-storage-users deployment isn't registering in the network map.

As I was writing this I also tried combining the storage-users and gateway services. I removed the opencloud storage-users server command and added the folowing env variables:

OC_RUN_SERVICES: gateway,storage-users
GATEWAY_STORAGE_USERS_MOUNT_ID #Configured the same as STORAGE_USERS_MOUNT_ID

I renamed the config file from storage-users.yaml to opencloud.yaml and restructured it as

gateway:
  loglevel: trace
storage-users:
  loglevel: trace
  ... # Rest is same as above

During startup I get the same error, but this time I get it from gateway and not storage-users

{"level":"info","service":"gateway","time":"2026-09-30T09:58:27Z","line":"github.com/opencloud-eu/opencloud/pkg/registry/register.go:19","message":"registering external service eu.opencloud.api.gateway-98ccde7c-df9c-44ed-aa83-e632687a26f7@127.0.0.1:9142"}
{"level":"fatal","service":"gateway","error":"Failed to connect to NATS Server: nats: no servers available for connection","time":"2026-09-30T09:58:27Z","line":"github.com/opencloud-eu/opencloud/pkg/registry/register.go:23","message":"Registration error for external service eu.opencloud.api.gateway"}

Weirdly enough, the connection is now there in Hubble even though is failing

NATS server seems to be healthy

{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"Starting nats-server"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Version:  2.14.0"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Git:      [not set]"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Cluster:  opencloud-cluster"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Name:     NCVHZKEQOZWWX47YRSGDMRJDY3EJWEYHORSI4LJRSUZGBSKZOKMTOBON"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Node:     dio6RPzV"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  ID:       NCVHZKEQOZWWX47YRSGDMRJDY3EJWEYHORSI4LJRSUZGBSKZOKMTOBON"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"Starting JetStream"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"    _ ___ _____ ___ _____ ___ ___   _   __  __"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":" _ | | __|_   _/ __|_   _| _ \\ __| /_\\ |  \\/  |"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"| || | _|  | | \\__ \\ | | |   / _| / _ \\| |\\/| |"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":" \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_|  |_|"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"         https://docs.nats.io/jetstream"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"---------------- JETSTREAM ----------------"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Strict:          true"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Max Memory:      94.10 GB"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Max Storage:     1.41 GB"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Store Directory: \"/var/lib/opencloud/nats/jetstream\""}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  API Level:       4"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"-------------------------------------------"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Starting restore for stream '$G > main-queue'"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Restored 0 messages for stream '$G > main-queue' in 7ms"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"  Recovering 10 consumers for stream - '$G > main-queue'"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"Took 64.93869ms to start JetStream"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"Listening for client connections on 0.0.0.0:9233"}
{"level":"info","service":"nats","time":"2026-09-30T10:02:13Z","line":"github.com/opencloud-eu/opencloud/services/nats/pkg/logging/nats.go:21","message":"Server is ready"}

Additional context

I really want to get this working as I'm desperately trying to get out of Nextcloud, I just don't know what's wrong with it. Been trying this with the rolling release (v8.0.1 and 7.2.0)

Navigation Menu

Repository navigation

Unable to connect to external nats (gateway/storage-users) #3623

Describe the bug

Steps to reproduce

Expected behavior

Actual behavior

Setup

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unable to connect to external nats (gateway/storage-users) #3623

Description

Describe the bug

Steps to reproduce

Expected behavior

Actual behavior

Setup

Additional context

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions