[Bug]: cnpg / barman cloud CNPG-I plugin failed to verify certificate: x509

Is there an existing issue already for this bug?

I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

1.26 (latest patch)

What version of Kubernetes are you using?

1.29

What is your Kubernetes environment?

Other

How did you install the operator?

OLM

What happened?

On Openshift Cluster on prem, v. 4.16, I installed the cpng-operator through OLM, and the barman plugin, according to the docs, through the manifests.

I created an objectstore like this:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: minio-store
spec:
  configuration:
    destinationPath: s3://cnpg-backups-rec/
    endpointURL: http://minioserver.testing.local:9000
    s3Credentials:
      accessKeyId:
        name: barman-minio
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: barman-minio
        key: ACCESS_SECRET_KEY
    wal:
      compression: gzip
  retentionPolicy: 30d

When I try to create a cluster using the barman cloud plugin, the cnpg operator does not create the pods and logs the lines I attached.

The barman-cloud pod exists and logs nothing at the cluster creation. I also checked that the barman-cloud certificates exists (client and server), they do.
When I try to create a cluster without specifying the barman cloud plugin, it works great and my cluster is created.
I installed both the cnpg operator and barman plugin in the namespace cnpg-system, and also tried to use the barman certificates to perform a curl request from another pod in the same namespace:

bash-5.1$ curl --cert barman_client.crt --key barman.key --cacert barman.crt -v https://barman-cloud:9090
*   Trying 172.30.247.153:9090...
* Connected to barman-cloud (172.30.247.153) port 9090 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: barman.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=barman-cloud
*  start date: Jun 30 13:27:36 2025 GMT
*  expire date: Sep 28 13:27:36 2025 GMT
*  subjectAltName: host "barman-cloud" matched cert's "barman-cloud"
*  issuer: CN=barman-cloud
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x563ba0254130)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/2
> Host: barman-cloud:9090
> user-agent: curl/7.76.1
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 415 
< content-type: application/grpc
< grpc-status: 3
< grpc-message: invalid gRPC request content-type ""
< 
* Connection #0 to host barman-cloud left intact

As a result, I'm not able to provide a cluster which is backuped. Am I missing something?

Cluster resource

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: somedb
spec:
  description: "test with somedb"
  imageCatalogRef:
    apiGroup: postgresql.cnpg.io
    kind: ClusterImageCatalog
    name: postgresql
    major: 15
  instances: 3
  startDelay: 300
  stopDelay: 300
  primaryUpdateStrategy: unsupervised
  bootstrap:
    initdb:
      database: somedb
      owner: somedb
  enableSuperuserAccess: true
  storage:
    storageClass: ocs-storagecluster-ceph-rbd
    size: 1Gi
  backup:
    plugins:
      - name: barman-cloud.cloudnative-pg.io
        isWalArchiver: true
        parameters:
          barmanObjectName: minio-store
  resources:
    requests:
      memory: "512Mi"
      cpu: "1"
    limits:
      memory: "1Gi"
      cpu: "2"

Relevant log output

{"level":"info","ts":"2025-07-01T13:37:24.560847807Z","msg":"Registered plugin","controller":"plugin","controllerGroup":"","controllerKind":"Service","Service":{"name":"barman-cloud","namespace":"cnpg-system"},"namespace":"cnpg-system","name":"barman-cloud","reconcileID":"401d2a85-d697-42bd-8436-3c0d02a1f1d0","pluginName":"barman-cloud.cloudnative-pg.io"}
{"level":"info","ts":"2025-07-01T13:38:08.358653761Z","logger":"cluster-resource","msg":"Defaulting for Cluster","version":"v1","name":"somedb","namespace":"testkda"}
{"level":"info","ts":"2025-07-01T13:38:08.372347316Z","logger":"cluster-resource","msg":"Validation for Cluster upon creation","version":"v1","name":"somedb","namespace":"testkda"}
{"level":"debug","ts":"2025-07-01T13:38:08.379543455Z","msg":"Reconciliation loop start","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"somedb","namespace":"testkda"},"namespace":"testkda","name":"somedb","reconcileID":"49e7aade-46d1-4c2c-9b09-1b377a25c49d","caller":"internal/controller/cluster_controller.go:152"}
{"level":"error","ts":"2025-07-01T13:38:08.432867144Z","logger":"setPluginProtocol","msg":"Error while connecting to plugin (logical)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"somedb","namespace":"testkda"},"namespace":"testkda","name":"somedb","reconcileID":"49e7aade-46d1-4c2c-9b09-1b377a25c49d","pluginName":"barman-cloud.cloudnative-pg.io","error":"while querying plugin identity: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/[email protected]/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/repository.(*data).setPluginProtocol.pluginConnectionConstructor.func1\n\tinternal/cnpi/plugin/repository/setup.go:100\ngithub.com/jackc/puddle/v2.(*Pool[...]).initResourceValue.func1\n\tpkg/mod/github.com/jackc/puddle/[email protected]/pool.go:423"}
{"level":"error","ts":"2025-07-01T13:38:08.433092027Z","msg":"Error loading plugins, retrying","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"somedb","namespace":"testkda"},"namespace":"testkda","name":"somedb","reconcileID":"49e7aade-46d1-4c2c-9b09-1b377a25c49d","error":"while getting plugin connection: while querying plugin identity: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/[email protected]/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/controller.(*ClusterReconciler).Reconcile\n\tinternal/controller/cluster_controller.go:210\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:255"}
{"level":"debug","ts":"2025-07-01T13:38:08.433174582Z","msg":"Reconciliation loop end","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"somedb","namespace":"testkda"},"namespace":"testkda","name":"somedb","reconcileID":"49e7aade-46d1-4c2c-9b09-1b377a25c49d","caller":"internal/controller/cluster_controller.go:154"}
{"level":"error","ts":"2025-07-01T13:38:08.433205711Z","msg":"Reconciler error","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"somedb","namespace":"testkda"},"namespace":"testkda","name":"somedb","reconcileID":"49e7aade-46d1-4c2c-9b09-1b377a25c49d","error":"while getting plugin connection: while querying plugin identity: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tpkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:255"}

Code of Conduct

I agree to follow this project's Code of Conduct

@kdautrey

Hi @kdautrey! I'm Dosu and I’m helping the cloudnative-pg team.

The error x509: certificate signed by unknown authority means the CNPG operator doesn't trust the CA that signed the barman-cloud server certificate. Even if curl works from another pod, the operator loads its own CA trust from a Kubernetes secret and uses that for plugin connections. If the CA in the secret doesn't match the one used by barman-cloud, you'll get this error.

Here's what to check:

Make sure the CA certificate used to sign the barman-cloud server certificate is the same one referenced by the CNPG operator for plugin trust. In your Cluster spec, you can explicitly set the serverCASecret under spec.certificates to point to the secret containing the correct CA (ca.crt) that signed barman-cloud's certs. If you don't set this, CNPG may generate its own self-signed CA, which won't match barman-cloud's CA, causing trust issues [code reference].
Double-check that the CA secret is in the right namespace and accessible to the operator. The secret must contain a valid ca.crt entry [code reference].
If you are using self-signed certs, ensure both the barman-cloud plugin and the CNPG operator are configured to use the same CA secret. You may need to manually create the secret and reference it in both places.
There have been recent changes to certificate handling and rotation (e.g., cert-manager integration), so if you are using automated certificate management, make sure the CA isn't being rotated or replaced unexpectedly [PR 7740].
If you use ArgoCD or similar tools, be aware of known issues where certificate bundles can disappear from webhooks during reconciliation, though this is less likely your root cause [issue 7597].

If you update your Cluster spec to explicitly reference the correct CA secret and ensure both the operator and barman-cloud trust the same CA, the plugin connection should succeed. If the problem persists, try enabling debug logs on the operator and check for any additional errors in the barman-cloud pod.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

@kdautrey

Hi @kdautrey, can you check if you're using the latest https://github.com/cloudnative-pg/plugin-barman-cloud (0.5.0 at the w.r.t) which updated the cnpg-i-machinery to take in the fix of cert reloading cloudnative-pg/cnpg-i-machinery#197

Hi!
Yes, I'm using this image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.5.0 since the beginning, and I'm facing the issue I described, I can't figure out why...

Update: this problem looks specific to the openshift community operator installed with OLM.

When I install the cnpg-operator with the manifests https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.26/releases/cnpg-1.26.1.yaml the secrets that are created are different:

cnpg-ca-secret: contains ca.crt and ca.key
cnpg-webhook-cert: contains tls.crt and tls.key. This certificate is signed by cnpg-ca-secret CA.

And when I create a cluster, I no longer have certificates trust issues between cnpg manager and the barman plugin (I have other issues related to a missing role though...).

When installed through OLM, I only have:

cnpg-controller-manager-service-cert: contains olmCAKey, tls.crt and tls.key

Maybe the barman plugin is not compatible with the OLM installation??

Hi. We are facing a similar issue, where the CNPG operator is not able to connect to the barman-cloud pod. What we noticed, is that the barman-cloud pod's certificate was rotated when the issue started. However, the issue doesn't seem to resolve itself without restarting the barman-cloud and cnpg operator pods, AND what's worse, is that the cnpg operator stops functioning while this issue persists. This just caused a downtime for us (switchover couldn't complete), and we had to restart the said pods manually in order to get the operator working again. This is a huge issue, that shouldn't happen in a production environment. I'm not sure if this issue is related or not, or if I should open a separate issue for it.

Operator log excerpt:

"msg":"Error loading plugins, retrying", "error":"while getting plugin connection: while querying plugin identity: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: remote error: tls: unknown certificate authority\""

@lgyurci

@lgyurci Same here, and in our case, we're not using OpenShift, just plain Kubernetes.

This issue is stale because it has been open for 60 days with no activity.

Can confirm, not using OpenShift, happened after the certificate got renewed, and it will keep on occuring every time it renews. Considering the CA/Browser Forum wants certificates to last no longer than 47 days (1), this means renewals will happen every month and this will become a pretty serious issue.

I can confirm this issue for OpenShift 4.18.23 and OLM installation of CloudNativePG v1.27.1: x509: certificate signed by unknown authority
When installing the same version via raw manifests the plugin works like intended.

Hello,
I can confirm the same issue happens on OpenShift 4.20.8 and using CloudNativePG v1.28.0.
As all the others there were no issues when isntalling the same version using directly the manifest provided in the Release section.

@kdautrey

Hi, @kdautrey. I'm Dosu, and I'm helping the cloudnative-pg team manage their backlog and am marking this issue as stale.

Issue Summary:

The CloudNativePG operator fails to create pods with the barman cloud plugin on OpenShift 4.16 due to x509 certificate verification errors.
Cluster creation without the plugin works fine, indicating the issue is specific to the plugin's certificate trust.
The root cause is the operator not trusting the CA that signed the barman-cloud server certificate, requiring explicit CA secret configuration.
Multiple users have reported similar certificate trust issues on OpenShift and Kubernetes, especially with OLM installations.
The problem is linked to certificate rotation and OLM's handling of CA secrets, causing operator downtime and manual pod restarts.

Next Steps:

Please confirm if this issue is still relevant with the latest version of CloudNativePG and your current environment.
If it is, you can keep the discussion open by commenting on this issue; otherwise, it will be automatically closed in 7 days.

Thank you for your understanding and contribution!

That AI is just hallucinating, the issue is that when certificates gets renewed the barman-cloud plugin doesn't handle it and will start to fail. It has nothing to do with OpenShift or OLM specifically, nor is it a problem with how pods are created or a missing CA secret configuration.

The temporary fix for this is to run kubectl -n cnpg-system rollout restart deployment barman-cloud when the certificate in question has been renewed, but the plugin should detect this itself and not require this manual step every month or so. With certificates starting to default to a 47 day lifespan this needs to be prioritized, instead it's been stale for a year.

I hit this trouble today and I will contribute rules because I received no alert too.

The issue still exists today, with openshift 4.20.15 and cnpg operator 1.29.1

I found the solution to my issue, in my particular case, when the operator is installed through OLM, if you have a global proxy setup in your openshift cluster, it injects HTTP_PROXY, HTTPS_PROXY, and NO_PROXY into every Deployment it manages, while the plain manifest does not! cnpg calls: "barman-cloud:9090", with no ".svc" or nothing, which goes straight to my proxy, and causes the issue.

Adding this to the subscription did the trick, it is working now!

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cloudnative-pg
  namespace: cnpg-system
spec:
  config:
    env:
    - name: HTTP_PROXY
      value: ""
    - name: HTTPS_PROXY
      value: ""
    - name: NO_PROXY
      value: ""

fix(plugin): update plugin address to use FQDN (#10921)

Navigation Menu

Repository navigation

[Bug]: cnpg / barman cloud CNPG-I plugin failed to verify certificate: x509 #7947

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: cnpg / barman cloud CNPG-I plugin failed to verify certificate: x509 #7947

Description

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Activity

dosubot commented on Jul 1, 2025

hh24k commented on Jul 21, 2025

kdautrey commented on Jul 28, 2025

kdautrey commented on Jul 29, 2025

lgyurci commented on Aug 13, 2025

dhess commented on Aug 20, 2025

github-actions commented on Oct 27, 2025

p3lim commented on Nov 3, 2025

st31n commented on Nov 20, 2025

jeremyalbrecht commented on Jan 14, 2026

dosubot commented on Apr 15, 2026

p3lim commented on Apr 15, 2026

davinkevin commented on Jun 9, 2026

kdautrey commented on Jun 16, 2026

kdautrey commented on Jun 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions