Node is not made primary when PVC is attached to new node

I am seeing that some resources are not made primary on the new node when a pod for example goes from node02 to node03.
The pod fails to mount the volume, causing it to be stuck in ContainerCreating:

MountVolume.SetUp failed for volume "pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7" : rpc error: code = Internal desc = NodePublishVolume failed for pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7: failed to set source device readwrite: exit status 1

Inside Kubernetes the volume attachment is correct:

apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  annotations:
    csi.alpha.kubernetes.io/node-id: node02
  creationTimestamp: "2026-01-19T01:04:40Z"
  finalizers:
  - external-attacher/linstor-csi-linbit-com
  name: csi-4ba15d6efb8bb7a7d9581a240d5bedd57b96b9a2bb7a873423da8182d05d0322
  resourceVersion: "31936834"
  uid: e74f54ad-e7d8-4b0b-9249-8f94113d643a
spec:
  attacher: linstor.csi.linbit.com
  nodeName: node02
  source:
    persistentVolumeName: pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7
status:
  attached: true
  attachmentMetadata:
    linstor.csi.linbit.com/device-path: /dev/drbd1017
    linstor.csi.linbit.com/fs-type: xfs
    linstor.csi.linbit.com/uses-publish-context: "true"

But the resource is still primary on node03, as indicated by the PVC annotation and drbdadm status:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: node03
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  creationTimestamp: "2026-01-11T23:30:00Z"
  finalizers:
  - kubernetes.io/pvc-protection
  - snapshot.storage.kubernetes.io/pvc-as-source-protection
  labels:
    kustomize.toolkit.fluxcd.io/name: apps-media
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: radarr-config
  namespace: media
  resourceVersion: "28820599"
  uid: 08c8e34a-ad2a-4394-ac5c-db0b48e927f7
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: pool1
  volumeMode: Filesystem
  volumeName: pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  phase: Bound

[root@node01:~]# drbdadm status pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7
pvc-08c8e34a-ad2a-4394-ac5c-db0b48e927f7 role:Secondary
  disk:Diskless open:no
  node02 role:Secondary
    peer-disk:UpToDate
  node03 role:Primary
    peer-disk:UpToDate

I can resolve this manually by making the current primary into secondary and then promoting the other node to primary.

Unsure if this is the LINSTOR project or part of Piraeus.

We generally use the DRBD "auto-promote" feature. That means that on using "mount /dev/drbd...", the DRBD device becomes primary, and on "umount /..." the device switches back to secondary.

I'm not aware of any bugs in DRBD in that area. However, if you manually promote a node to Primary (using drbdadm primary ...) you then also have to manually run drbdadm secondary ..... Manual promotion disables the auto-promote feature.

So if you manually promote a resource, you kind of break the CSI driver, as it then does not automatically demote after unmounting.

We generally use the DRBD "auto-promote" feature. That means that on using "mount /dev/drbd...", the DRBD device becomes primary, and on "umount /..." the device switches back to secondary.

I'm not aware of any bugs in DRBD in that area. However, if you manually promote a node to Primary (using drbdadm primary ...) you then also have to manually run drbdadm secondary ..... Manual promotion disables the auto-promote feature.

So if you manually promote a resource, you kind of break the CSI driver, as it then does not automatically demote after unmounting.

I did not know that it breaks the promotion. It's possible I have messed around with it while creating it and that is why it's broken, I'm not seeing that issue for other volumes (only this one and another).

Is there a way to fix it, eg. can I set auto-promote on the resource again?

Just tried setting the auto-promote option on the resource definition, but no luck. It gets set but has no effect on the issue.

linstor resource-definition opt pvc-95e7ee1f-dcae-4832-ba69-16c7d95d6079 --auto-place=yes

Checked if the current primary still has it mounted somehow, but it doesnt appear so when checking with lsblk | grep drbd1013.

I don't think there is a command that can reset this flag, sadly.

I don't think there is a command that can reset this flag, sadly.

Would it then be best to create this an issue to LINSTOR server for a possible solution from their side?

It seems like I am forced to delete the whole resource/PVC and create it again with data from backup to get it to act normally again.

It seems like I am forced to delete the whole resource/PVC and create it again with data from backup to get it to act normally again.

No need to go to those extremes. What you need to to:

Ensure that the volume is not mounted anywhere. So basically: remove / scale down the Pods using the volume.
Check in LINSTOR: is the volume marked InUse?
If yes: Run drbdadm secondary ... from the satellite.
Now, you can start the Pods again

What I meant with the flag not resetting is that when you run drbdadm primary ... you can not get the auto-demote until you manually ran drbdadm secondary ... again after all mounts are gone.

I totally misunderstood you, apologies.
Doing as you said solves my issue.

We are hitting this exact behavior on a freshly provisioned PVC in
Cozystack e2e CI runs (Talos sandbox, kernel 6.x, drbd-utils 9.x,
piraeus-operator from current cozystack chart).

Repro is more aggressive than the original report — no pod migration
is involved. The PVC is provisioned, mounted exactly once, and ends
up stuck because the LINSTOR resource was promoted to Primary on a
node that never had a consumer.

Setup

3 nodes (srv1/srv2/srv3), 3 replicas, placementCount: 3
StorageClass replicated, fresh vlstorage-generic-0 StatefulSet pod
Pod was scheduled on srv1 at creation time and never moved
(pod UID unchanged for 28+ minutes)

State

$ kubectl get volumeattachment ... -o json
spec.nodeName: srv1
status.attached: true

$ drbdsetup status pvc-433a983f-...
pvc-433a983f-... role:Primary
  disk:UpToDate open:no            # <-- nobody holds the device
  srv1 role:Secondary
    peer-disk:UpToDate
  srv2 role:Secondary
    peer-disk:UpToDate

LINSTOR view:

| pvc-433a983f-... | srv1 | Unused | Ok | UpToDate |
| pvc-433a983f-... | srv2 | Unused | Ok | UpToDate |
| pvc-433a983f-... | srv3 | InUse  | Ok | UpToDate |   <- stuck Primary on a node that has no pod

CSI on srv1 retries NodePublishVolume every ~2 minutes, each time
failing with the canonical:

NodePublishVolume failed for pvc-433a983f-...:
failed to set source device readwrite: exit status 1

Why this matters

The previous report was closed because the drbdadm secondary
workaround restored access. But the underlying race is still there
and trips up CI deterministically: it is reproducible on a fresh
sandbox where no pod ever migrated and no failover ever happened.

A volume that comes up Primary on a node that never had a consumer
indicates auto-promote firing during the provisioning path
(mkfs.ext4 against a freshly created replica) without the matching
auto-demote when the path closes. Once that flag is set, only an
explicit drbdadm secondary resets it, and there is no automation
for that.

Closely related: piraeusdatastore/linstor-csi#172 (open) describes
the same symptom on bulk PVC creation.

Workaround that worked for us

kubectl -n cozy-linstor exec linstor-satellite.<node> -c linstor-satellite -- \
  drbdadm secondary pvc-<id>

CSI on the target node then promotes locally on the next retry and
the pod attaches.

Asks

Could the operator (or linstor-satellite) detect "Primary with
open:no AND no matching VolumeAttachment on this node" and demote
automatically? open:no makes this a safe operation.
Or: could the provisioning path (mkfs) be wrapped to guarantee
drbdadm secondary on close, regardless of how the path exits?

Reopening because the upstream code state is unchanged and the issue
keeps recurring across users.

fix(linstor): plunger demotes idle-Primary DRBD resources (#2539)

fix(linstor): plunger demotes idle-Primary DRBD resources

Navigation Menu

Repository navigation

Node is not made primary when PVC is attached to new node #942

Setup

State

Why this matters

Workaround that worked for us

Asks

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Node is not made primary when PVC is attached to new node #942

Description

Activity

WanzenBug commented on Jan 20, 2026

mikkel1156 commented on Jan 20, 2026

mikkel1156 commented on Jan 21, 2026

WanzenBug commented on Jan 22, 2026

mikkel1156 commented on Jan 22, 2026

WanzenBug commented on Jan 22, 2026

mikkel1156 commented on Jan 22, 2026

kvaps commented on Apr 29, 2026

Setup

State

Why this matters

Workaround that worked for us

Asks

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions