rbd: fix healer staging path for Block volumeMode by YLShiJustFly · Pull Request #6257 · ceph/ceph-csi

YLShiJustFly · 2026-04-29T08:27:55Z

rbd: fix healer staging path for Block volumeMode

Block volumeMode PVCs using rbd-nbd mounter permanently lose IO after
CSI plugin pod restart due to two bugs in rbd_healer.go:

formatStagingTargetPath computes a filesystem-format path
(.../csi///globalmount) for all volumes, but Block
volumes use a completely different path:
.../csi/volumeDevices/staging//
ValidateNodeStageVolumeRequest calls checkDirExists before any healer
logic, so the wrong path causes an immediate InvalidArgument error and
attachRBDImage never runs.
callNodeStageVolume detects Block volumes using
CSI.FSType == "block", but Block volumeMode PVs have an empty FSType.
This is always false, causing the healer to send a Mount capability
instead of a Block capability.

Filesystem PVCs are unaffected because their sha256 path is correct.
Block PVCs have no fallback recovery path (kubelet does not re-call
NodeStageVolume when the bind mount is still in mountinfo), so IO is
permanently lost until the node is restarted.

Fix by:

Adding a Block branch to formatStagingTargetPath using pv.Name as the
staging subdirectory, matching kubelet actual path layout
Changing Block detection to use pv.Spec.VolumeMode
Removing the unused fsTypeBlockName constant

nixpanic · 2026-05-01T08:47:55Z

Thanks for the PR! Please correct the subject of the commit, so that it says:

rbd: fix healer staging path and capability type for Block volumeMode

Ideally an e2e test is added for this as well. Please check e2e/rbd.go for a similar test that uses rbd-nbd and a filesystem volume,

YLShiJustFly · 2026-05-02T01:53:04Z

Thanks for the review! I've addressed both points:

Fixed the
commit subject to
rbd: fix healer staging path and capability type for Block volumeMode
Added an e2e test
perform IO on rbd-nbd Block volume after nodeplugin restart
in e2e/rbd. go, following the pattern of the existing filesystem volume healer test

iPraveenParihar · 2026-06-02T10:31:18Z

@Mergifyio rebase

iPraveenParihar · 2026-06-02T10:31:18Z

@Mergifyio rebase

mergify · 2026-06-02T10:31:57Z

rebase

☑️ Command `rebase` ignored because it is already running from a previous command.

mergify · 2026-06-02T10:32:38Z

Deprecation notice: This pull request comes from a fork and was rebased using bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the rebase action will no longer be able to rebase fork pull requests with this configuration. Please switch to the update action/command to ensure compatibility going forward.

mergify · 2026-06-02T10:32:38Z

rebase

✅ Branch has been successfully rebased

iPraveenParihar · 2026-06-02T10:45:58Z

/test ci/centos/mini-e2e/k8s-1.35

nixpanic · 2026-06-24T14:38:14Z

@Mergifyio rebase

mergify · 2026-06-24T14:38:28Z

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

Block volumeMode PVCs using rbd-nbd mounter permanently lose IO after CSI plugin pod restart due to two bugs in rbd_healer.go: 1. formatStagingTargetPath computes a filesystem-format path (.../csi/<driver>/<sha256>/globalmount) for all volumes, but Block volumes use a completely different path: .../csi/volumeDevices/staging/<pv-name>/ ValidateNodeStageVolumeRequest calls checkDirExists before any healer logic, so the wrong path causes an immediate InvalidArgument error and attachRBDImage never runs. 2. callNodeStageVolume detects Block volumes using CSI.FSType == "block", but Block volumeMode PVs have an empty FSType. This is always false, causing the healer to send a Mount capability instead of a Block capability. Filesystem PVCs are unaffected because their sha256 path is correct. Block PVCs have no fallback recovery path (kubelet does not re-call NodeStageVolume when the bind mount is still in mountinfo), so IO is permanently lost until the node is restarted. Fix by: - Adding a Block branch to formatStagingTargetPath using pv.Name as the staging subdirectory, matching kubelet actual path layout - Changing Block detection to use pv.Spec.VolumeMode - Removing the unused fsTypeBlockName constant Add an e2e test that verifies IO on a Block volumeMode PVC using rbd-nbd mounter is restored after nodeplugin restart. Signed-off-by: YLShiJustFly <[email protected]>

ceph-csi-bot · 2026-06-24T14:38:52Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-06-24T14:38:53Z

/test ci/centos/k8s-e2e-external-storage/1.36

ceph-csi-bot · 2026-06-24T14:38:53Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-06-24T14:38:54Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-06-24T14:38:54Z

/test ci/centos/mini-e2e-helm/k8s-1.36

ceph-csi-bot · 2026-06-24T14:38:54Z

/test ci/centos/mini-e2e/k8s-1.35

ceph-csi-bot · 2026-06-24T14:38:54Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-06-24T14:38:54Z

/test ci/centos/mini-e2e/k8s-1.36

ceph-csi-bot · 2026-06-24T14:38:55Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-06-24T14:38:55Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-06-24T14:38:56Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2026-06-24T15:02:44Z

/retest ci/centos/k8s-e2e-external-storage/1.34

nixpanic · 2026-06-24T15:03:03Z

/retest ci/centos/k8s-e2e-external-storage/1.34

Deployment failed logs

YLShiJustFly · 2026-06-25T03:29:01Z

Hi @nixpanic,

Just a gentle reminder regarding this PR. It has already received 2 approvals.

Regarding the current CI failures:

The k8s-1.34 failure is the exact same JSON unmarshaling error (cannot unmarshal string into Go value of type []DeleterConfig) that occurred in journal: lower omap not found log level in getOMapValuesByKeys #6356, which is a broken upstream infra issue.
The k8s-1.36 failure is a known PodSecurity timeout.

Since these failures are completely unrelated to this RBD healer patch (which only refactors the internal path string formatting), could you please help apply the 'approved' label to bypass the CI and manually merge this, just like #6356?

Thank you for your time and help!

iPraveenParihar · 2026-06-25T04:14:11Z

/retest ci/centos/mini-e2e/k8s-1.35

iPraveenParihar · 2026-06-25T04:14:20Z

/retest ci/centos/mini-e2e/k8s-1.36

iPraveenParihar · 2026-06-25T06:29:05Z

/retest ci/centos/mini-e2e/k8s-1.36

mergify · 2026-06-25T09:22:35Z

Queued — the merge queue status continues in this comment ↓.

mergify · 2026-06-25T09:22:38Z

Deprecation notice: This pull request comes from a fork and was queued with update_method=rebase and update_bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the merge queue will no longer be able to rebase fork pull requests with this configuration. To avoid disruption, switch to update_method=merge in your queue rule.

mergify · 2026-06-25T09:22:40Z

Merge Queue Status

✅ Entered queue — 2026-06-25 09:22 UTC · Rule: default · triggered by merge protections
✅ Checks skipped · PR is already up-to-date
✅ Merged — 2026-06-25 09:22 UTC · at f6f9c9d801bb276705c0e22ecfd9b3815d7a9ede · rebase

This pull request spent 11 seconds in the queue, including 2 seconds running CI.

Required conditions to merge

#approved-reviews-by >= 2 [🛡 GitHub branch protection]
- rbd: fix healer staging path for Block volumeMode #6257

mergify Bot added the bug Something isn't working label Apr 29, 2026

YLShiJustFly force-pushed the fix/healer-block-staging-path branch from 7575ab1 to 79e58f3 Compare May 2, 2026 01:42

YLShiJustFly changed the title ~~fix(rbd): fix healer staging path and capability type for Block volumeMode~~ rbd: fix healer staging path and capability type for Block volumeMode May 2, 2026

mergify Bot added the component/rbd Issues related to RBD label May 2, 2026

YLShiJustFly force-pushed the fix/healer-block-staging-path branch from 79e58f3 to 717e127 Compare May 11, 2026 09:01

YLShiJustFly changed the title ~~rbd: fix healer staging path and capability type for Block volumeMode~~ rbd: fix healer staging path for Block volumeMode May 11, 2026

ceph-csi-bot force-pushed the fix/healer-block-staging-path branch from 717e127 to 3b6fced Compare June 2, 2026 10:32

iPraveenParihar requested a review from a team June 17, 2026 12:06

Madhu-1 approved these changes Jun 23, 2026

View reviewed changes

Madhu-1 requested a review from iPraveenParihar June 23, 2026 05:17

iPraveenParihar approved these changes Jun 23, 2026

View reviewed changes

ceph-csi-bot force-pushed the fix/healer-block-staging-path branch from 3b6fced to f6f9c9d Compare June 24, 2026 14:38

nixpanic added the ok-to-test Label to trigger E2E tests label Jun 24, 2026

ceph-csi-bot added ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. and removed ok-to-test Label to trigger E2E tests labels Jun 24, 2026

mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Jun 24, 2026

mergify Bot merged commit 9dac64d into ceph:devel Jun 25, 2026
41 of 42 checks passed

Conversation

YLShiJustFly commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YLShiJustFly commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iPraveenParihar commented Jun 2, 2026

Uh oh!

iPraveenParihar commented Jun 2, 2026

Uh oh!

mergify Bot commented Jun 2, 2026

☑️ Command rebase ignored because it is already running from a previous command.

Uh oh!

mergify Bot commented Jun 2, 2026

Uh oh!

mergify Bot commented Jun 2, 2026

✅ Branch has been successfully rebased

Uh oh!

iPraveenParihar commented Jun 2, 2026

Uh oh!

nixpanic commented Jun 24, 2026

Uh oh!

mergify Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request rule doesn't match anymore

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

ceph-csi-bot commented Jun 24, 2026

Uh oh!

nixpanic commented Jun 24, 2026

Uh oh!

nixpanic commented Jun 24, 2026

Uh oh!

YLShiJustFly commented Jun 25, 2026

Uh oh!

iPraveenParihar commented Jun 25, 2026

Uh oh!

iPraveenParihar commented Jun 25, 2026

Uh oh!

iPraveenParihar commented Jun 25, 2026

Uh oh!

mergify Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Jun 25, 2026

Uh oh!

mergify Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

YLShiJustFly commented Apr 29, 2026 •

edited

Loading

nixpanic commented May 1, 2026 •

edited

Loading

YLShiJustFly commented May 2, 2026 •

edited

Loading

☑️ Command `rebase` ignored because it is already running from a previous command.

mergify Bot commented Jun 24, 2026 •

edited

Loading

mergify Bot commented Jun 25, 2026 •

edited

Loading

mergify Bot commented Jun 25, 2026 •

edited

Loading