Skip to content

nvmeof: simplify subsystem deletion - #6533

Merged
mergify[bot] merged 1 commit into
ceph:develfrom
gadididi:nvmeof/simplify_delete_subsystem
Sep 8, 2026
Merged

mergify[bot] merged 1 commit into
ceph:develfrom
gadididi:nvmeof/simplify_delete_subsystem

Conversation

@gadididi

@gadididi gadididi commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

instead of call 4 GRPc calls to the nvmeof gw:

  1. SubsystemExists
  2. ListNamespaces  - this call takes really more time.
  3. then it delete per listener DeleteListener
  4. then finally delete the subsystem -  DeleteSubsystem

jsut call DeleteSubsystem, if subsystem not empty
will retrun EBUSY.
if subsystem does not exist, return ENOENT.
the GW automatically deletes the
listeners.

UPDATE:
tested manually on ODF cluster with 2 PVCs.

  1. create pvc1
  2. create pvc2
  3. delete pvc1 -> see this error (from GW)
I0903 09:43:03.516339       1 controllerserver.go:953] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 Deleting namespace 1 for subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:43:03.516347       1 nvmeof.go:157] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 Deleting namespace 1 from subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:43:03.772863       1 nvmeof.go:170] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 Namespace deleted successfully: 1
I0903 09:43:03.772893       1 controllerserver.go:958] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 Namespace 1 deleted for subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:43:03.772901       1 nvmeof.go:407] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 Deleting NVMe subsystem: nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:43:03.775376       1 nvmeof.go:422] ID: 17 Req-ID: 0001-0011-openshift-storage-0000000000000002-6efc92bb-024a-44ab-ad06-12147a2233d2 subsystem nqn.2016-06.io.ceph:subsystem.test-integration has namespaces and cannot be deleted: Failure deleting subsystem nqn.2016-06.io.ceph:subsystem.test-integration: Namespace 2 is still using the subsystem. Either remove it or use the "--force" command line option

we handle it by EBUSY

  1. delete pvc1 -> success and subsystem were deleted.
I0903 09:44:37.716509       1 controllerserver.go:953] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Deleting namespace 2 for subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:44:37.716515       1 nvmeof.go:157] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Deleting namespace 2 from subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:44:37.934015       1 nvmeof.go:170] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Namespace deleted successfully: 2
I0903 09:44:37.934043       1 controllerserver.go:958] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Namespace 2 deleted for subsystem nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:44:37.934050       1 nvmeof.go:407] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Deleting NVMe subsystem: nqn.2016-06.io.ceph:subsystem.test-integration
I0903 09:44:37.978557       1 nvmeof.go:429] ID: 19 Req-ID: 0001-0011-openshift-storage-0000000000000002-a675552c-1e8f-4339-a884-cf417b379105 Subsystem deleted successfully: nqn.2016-06.io.ceph:subsystem.test-integration

Get CI jobs structured, wait for others to finish before starting more.

Depends-on: #6526 #6515

@gadididi
gadididi requested a review from nixpanic September 3, 2026 09:25
@gadididi gadididi self-assigned this Sep 3, 2026
@gadididi
gadididi requested a review from a team as a code owner September 3, 2026 09:25
Copilot AI lite review requested due to automatic review settings September 3, 2026 09:25
@gadididi
gadididi requested a review from a team as a code owner September 3, 2026 09:25
@gadididi
gadididi requested review from Madhu-1 and removed request for a team and Copilot September 3, 2026 09:25
@mergify mergify Bot added the component/nvme-of Issues and PRs related to NVMe-oF. label Sep 3, 2026
@Madhu-1
Madhu-1 requested a balanced review from Copilot September 3, 2026 10:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

DeleteSubsystem currently hides EBUSY, changing its contract for all callers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Simplifies NVMe-oF subsystem cleanup by relying on gateway deletion semantics.

Changes:

  • Removes namespace/listener preflight calls.
  • Treats ENOENT and EBUSY as expected outcomes.
File summaries
File Review
internal/nvmeof/nvmeof.go Must preserve EBUSY as a typed/sentinel error and handle it only in cleanup.
internal/nvmeof/controller/controllerserver.go Cleanup is simplified; update the inaccurate error-handling comment.
Review details

Suppressed comments (1)

internal/nvmeof/nvmeof.go:425

  • Please add an automated unit test for the new status handling, especially that EBUSY follows the intended path while unrelated nonzero statuses still fail. This errno branch is the core behavior of the simplification, but the existing tests do not exercise DeleteSubsystem response statuses.
	case status.GetStatus() == int32(syscall.EBUSY):
		log.DebugLog(ctx, "subsystem %s has namespaces and cannot be deleted: %s", subsystemNQN, status.GetErrorMessage())

		return nil // Treat as success for our use case, cannot delete subsystem because it is busy
	case status.GetStatus() != 0:
  • Files reviewed: 2/2 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/nvmeof/nvmeof.go Outdated
Comment on lines +421 to +424
case status.GetStatus() == int32(syscall.EBUSY):
log.DebugLog(ctx, "subsystem %s has namespaces and cannot be deleted: %s", subsystemNQN, status.GetErrorMessage())

return nil // Treat as success for our use case, cannot delete subsystem because it is busy

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only one calls for this function is nvmeof-csi driver, and we want to handle as no-error the case there are more ns in this subsystem

Comment thread internal/nvmeof/controller/controllerserver.go Outdated
Comment thread internal/nvmeof/nvmeof.go
log.DebugLog(ctx, "Subsystem %s already deleted (not found)", subsystemNQN)

return nil
case status.GetStatus() == int32(syscall.EBUSY):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is success becase we have a namespace in it but it will get deleted successfully when no namespace exists? can you please add more comment for this case?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add.
this case when delete the subsystem failed from the GW side because there is more namespace in this subsystem (not the one we just removed in the previous call deleteNamespace, other)

I added this log.DebugLog(ctx, "subsystem %s has namespaces and cannot be deleted: %s", subsystemNQN, status.GetErrorMessage())

I will add also comment explanation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed to return new error, and in the nvmeof controller I handle it if errors.Is(err, nvmeoferrors.ErrSubsystemHasNamespaces)

Comment thread internal/nvmeof/controller/controllerserver.go
@gadididi
gadididi force-pushed the nvmeof/simplify_delete_subsystem branch from 3d53ac6 to e584014 Compare September 3, 2026 11:21
@gadididi
gadididi requested a review from Madhu-1 September 3, 2026 11:22
Comment thread internal/nvmeof/nvmeof.go
case status.GetStatus() == int32(syscall.EBUSY):
log.DebugLog(ctx, "subsystem %s has namespaces and cannot be deleted: %s", subsystemNQN, status.GetErrorMessage())

return fmt.Errorf("%w: %s", nvmeoferrors.ErrSubsystemHasNamespaces, status.GetErrorMessage())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this is not required we should follow same pattern we are following for ENOENT,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think returning an error is nice. The function did not do what was requested (DeleteSubsystem()), and this gives a good way to handle the error in different ways.

@nixpanic
nixpanic requested review from a team and Madhu-1 September 4, 2026 12:32
@nixpanic

nixpanic commented Sep 4, 2026

Copy link
Copy Markdown
Member

This is a great performance enhancement, the listing of namespaces takes quite long, and is very visible when deleting a volume. Without the listing, DeleteVolume is much quicker!

@nixpanic

nixpanic commented Sep 5, 2026

Copy link
Copy Markdown
Member

/queue

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

@Mergifyio rebase

@mergify

mergify Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

@ceph-csi-bot
ceph-csi-bot force-pushed the nvmeof/simplify_delete_subsystem branch from e584014 to f4ee275 Compare September 5, 2026 10:48
@ceph-csi-bot ceph-csi-bot added the ok-to-test Label to trigger E2E tests label Sep 5, 2026
@nixpanic nixpanic removed the dequeued label Sep 8, 2026
@mergify mergify Bot added the dequeued label Sep 8, 2026
@nixpanic

nixpanic commented Sep 8, 2026

Copy link
Copy Markdown
Member

/queue

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

@Mergifyio rebase

@mergify

mergify Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

instead of call 4 GRPc calls to the nvmeof gw:
1. SubsystemExists
2. ListNamespaces  - this call takes really more time.
3. then it delete per listener DeleteListener
4. then finally delete the subsystem -  DeleteSubsystem

jsut call DeleteSubsystem, if subsystem not empty
will retrun `EBUSY`.
if subsystem does not exist, return `ENOENT`.
the GW automatically deletes the
listeners.

Signed-off-by: gadi-didi <[email protected]>
@ceph-csi-bot
ceph-csi-bot force-pushed the nvmeof/simplify_delete_subsystem branch from 736ee4f to d919a05 Compare September 8, 2026 11:01
@ceph-csi-bot ceph-csi-bot added ok-to-test Label to trigger E2E tests and removed queued/rebase labels Sep 8, 2026
@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/upgrade-tests-cephfs

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.36

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/upgrade-tests-rbd

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.35

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.35

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.36

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/k8s-e2e-external-storage/1.34

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e/k8s-1.35

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e/k8s-1.36

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e-helm/k8s-1.34

@ceph-csi-bot

Copy link
Copy Markdown
Collaborator

/test ci/centos/mini-e2e/k8s-1.34

@ceph-csi-bot ceph-csi-bot added the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Sep 8, 2026
@mergify mergify Bot removed the dequeued label Sep 8, 2026
@ceph-csi-bot ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Sep 8, 2026
@mergify mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Sep 8, 2026
@iPraveenParihar

Copy link
Copy Markdown
Contributor

/retest ci/centos/mini-e2e/k8s-1.34

@mergify

mergify Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • ✅ Entered queue — 2026-09-08 14:54 UTC · Rule: default · triggered by merge protections
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged — 2026-09-08 14:55 UTC · at de6deffa48e885ad0d24f9dd8efd5f315333da9f · rebase

This pull request spent 21 seconds in the queue, including 2 seconds running CI.

Required conditions to merge

@mergify
mergify Bot merged commit de6deff into ceph:devel Sep 8, 2026
45 of 46 checks passed
@gadididi
gadididi deleted the nvmeof/simplify_delete_subsystem branch September 9, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/nvme-of Issues and PRs related to NVMe-oF.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants