unhealthy is also used for volumes, maybe call it disfunctional?

disconnected isn't really correct, as a disconnected client can not reach the storage. A client may re-connect later on, causing havoc. Or, the node may be somehow broken, but apps may still be running, this is the major concern.

csi.storage.k8s.io isn't appropriate, use csi.ceph.io or something similar instead

Ensure that this is not copied to the clones and snapshots.

How about below

• For RBD using something like .rbd.csi.ceph.com so that we dont need to worry about the replication but we still need to take care of clone/snapshots
• For cephfs use .cephfs.csi.ceph.com ?

➕ .rbd.csi.ceph.com and .cephfs.csi.ceph.com will be consistent with other metadata key as well.

how?

Ensure that this is not copied to the clones and snapshots.

This steps is missing the details to get the clientAdress+nonce.

@Madhu-1, does this get us the required address?

it will give us the IP address not nonce

Ahh okay, from comment I thought we get the IP+nonce

For RBD, we can get the clientAddres+nonce from below node path

[root@c1-m03 /]# cat /sys/devices/rbd/0/client_addr
192.168.39.34:0/2458835906

suggested by @nixpanic

@Madhu-1, does this get us the required address?

ceph-csi/internal/util/connection.go

Lines 154 to 163 in cfb3979

	// GetAddrs returns the addresses of the RADOS session,
	// suitable for blocklisting.
	func (cc *ClusterConnection) GetAddrs() (string, error) {
	if cc.conn == nil {
	return "", errors.New("cluster is not connected yet")
	}
	return cc.conn.GetAddrs()
	}

I tried this, but it returns a different nonce, IP being same.
The method mentioned earlier in previous comment only applies to block devices. (/sys/devices/rbd/0/client_addr).
I haven't found any other way to retrieve the IP and nonce together except calling rbd status/ceph tell API calls.

That said, I don't see any difference between blocklisting by just IP vs IP+nonce. Since the primary goal is to block the access from the tainted node, so blocklisting by IP alone should suffice?

^^ @Madhu-1 @nixpanic @Rakshith-R

Are we not going to store it in the metadata?

we could store the clientId for CephFs subvolumes, that would add one extra metadata entry but could save us from listing the active clients and matching operation for each subvolume request.

https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#non-graceful-node-shutdown says that node.kubernetes.io/out-of-service taint should be added only if "the node is already in shutdown or power off state (not in the middle of restarting)". How can there be active client sessions and applications still be running there if the node is physically powered off?

@idryomov we are trying to cover the worst case where we lost access to the nodes and we want to block the client on that node to avoid the problems to be on safer side. When the taint is added the kubernetes assumes that node is already in power off state and starts moving the work load to other nodes, we are trying to have more fencing mechanism to ensure that we are fencing off the node to avoid the access to the client (if they run or ever comes back when the taint still exists)

What about the point when the taint is removed? Since the admin can apply the taint without knowing that the node is actually powered off (and that being precisely what this design is trying to guard against), I'm wondering whether the admin can also remove the taint whenever they feel like it?

Put differently, is it enforced that the node goes through (an equivalent of) a power cycle before the taint is removed?

When the node is recovered and admin can access the node, If the node is available (kubelet+api-server) ensure that all pods are removed from the node (to keep it consistent with the ETCD data of pods) and no new pods are scheduled on the node.

When everything is fine the admin can remove the taint and mark the node available for new pods scheduling. The expectation is that the admin will remove the taints when there are no pods running on the node. As we are implementing this in the RPC spec. CSI takes care of unfencing (when a pod is scheduled for the node) and allow the future operations.

Does the term "recovered" include a power cycle?

No pods running on the node is very different from (an equivalent of) a power cycle. Let's say the taint gets added on an active/running node with a bunch of mapped RBD devices and the corresponding mounts. Later, the admin realizes that they screwed up and decides to remove the taint. What is supposed to ensure that those mounts are teared down before Ceph CSI unfences?

The kubelet will take care of it as soon the node comes back.

Can you go into more detail here? How exactly does kubelet do that? If one of the steps there fails for some reason, is the admin prevented from removing the taint?

And later its admin responsibility to power of the node as well.

... but this isn't enforced?

The kubelet will take care of it as soon the node comes back.

Can you go into more detail here? How exactly does kubelet do that? If one of the steps there fails for some reason, is the admin prevented from removing the taint?

I havent looked into the kubelet code but we can check on that one @iPraveenParihar PTAL. Not yet, kubernetes doesnt have any checks before removing the taint, its left to the admin to validate everything and remove the taint (we can also document if anything is required from our side as well)

And later its admin responsibility to power of the node as well.

... but this isn't enforced?

yes nothing is enforced in kubernetes its only documented step.

From what I have tested...

Let's say the taint gets added on an active/running node with a bunch of mapped RBD devices and the corresponding mounts. Later, the admin realizes that they screwed up and decides to remove the taint. What is supposed to ensure that those mounts are teared down before Ceph CSI unfences?

Scenario 1: Node is healthy (Node is Ready, kubelet is running), taints are removed without reboot.

• If a taint is added to a healthy node where the kubelet is functional, existing pods are marked for deletion but stay in a Terminating or Error state.

• These pods aren’t forcefully deleted because the node's status is not NotReady.

• NodeUnpublishVolume and NodeUnstageVolume are not called, since the CSI nodeplugin pods are also removed.

• ControllerUnpublishVolume is not triggered either — VolumeAttachment objects stay intact until pods are forcefully deleted.

• New pods scheduled to another node will fail with attach errors due to the image still being mapped.

• The admin realizes that they screwed up and decides to remove the taint, nodeplugin pods come back up, kubelet issues the pending NodeUnpublishVolume/NodeUnstageVolume calls for the pod/s in terminating/error state, mappings are removed, and new pods on the other node can finally attach the volume successfully.

I'm trying to explore the scenario described in this doc -- one where NodeUnstageVolume and NodeUnpublishVolume calls either aren't issued at all or don't make it and only ControllerUnpublishVolume goes in effect. If the node isn't powered off, the mounts would still be there and would be left behind, right? My understanding is that only blocklisting would occur as part of handling ControllerUnpublishVolume.

Scenario 2: Node is unresponsive (Node is NotReady, kubelet is not running), taints are removed without reboot

• The unresponsive node is tainted.

• Pods are forcefully deleted.

• VolumeAttachment deletion triggers ControllerUnpublishVolume, which blocks the RBD image.

• New pods are scheduled on other nodes and can attach without issue.

• Here, now if the node comes back without a reboot and Kubelet is running, NodeUnpublishVolume/NodeUnstageVolume would not be called as the pod/s is already deleted. Since the node hasn’t rebooted, the original RBD device mappings and mount points still persist.

The kubelet will take care of it as soon the node comes back. And later its admin responsibility to power of the node as well. i hope @iPraveenParihar already tested this case as well.

@Madhu-1, the mounts and mappings will persist unless the node is rebooted. That’s why we discussed that the admin should only remove the taint after the node has come back from a successful shutdown. This ensures clean teardown and avoids stale device state.

Here, now if the node comes back without a reboot and Kubelet is running, NodeUnpublishVolume/NodeUnstageVolume would not be called as the pod/s is already deleted. Since the node hasn’t rebooted, the original RBD device mappings and mount points still persist.

This is exactly what I suspected and why I kept probing here ;) This should be obvious by now but I want to restate it just in case: if the node a) isn't properly cleaned up (i.e. all mounts and RBD device mappings are teared down via NodeUnpublishVolume/NodeUnstageVolume) or b) doesn't go through a power cycle to waive any concerns around cleanup, removing the blocklist entry in ControllerPublishVolume can lead to data corruption. This was added after my earlier comment

       # The blocklist must persist until we can confirm the node has gone through
       # a complete power cycle, as premature expiration could lead to data corruption

but not in the main body of the doc and previously wasn't mentioned at all. I'd suggest highlighting this in a separate paragraph.

Thanks @idryomov for probing on this one to ensure we dont miss anything and we document it :) , @iPraveenParihar Thanks for confirming as my memory is really old for this as this exists for many releases and i tested very long back. We need to document this in our documents what is the exception from the ceph-csi to avoid any problems related to data.

A "problem" with OSD blocklist entries is that by default they expire after 1 hour. The expiration timeout is in seconds and can be changed by passing an additional integer argument. For example, ceph osd blocklist add 1.2.3.4/1234 86400 would persist the blocklist entry for 1 day.

If the goal here is to accommodate scenarios where the admin applies the out-of-service taint without knowing that the node is actually powered off, the blocklist entry needs to persist until it becomes known for sure that the node went through (an equivalent of) a power cycle. Allowing the blocklist entry to expire before that point can lead to "stale client access" and therefore data corruption.

we will blocklist of the max period (like 5 years) or indefinite time. @iPraveenParihar can you please update the time for that as well.

just for referenece:

(via NodeUnstageVolumeandNodeUnpublishVolume) to (via NodeUnpublishVolumeandNodeUnstageVolume)

remove the nonce as we will go with ip blocklisting?

add rbd namespace as well to the command

why we have csi.ceph.io?

mistake should be csi.storage.k8s.io/controller-publish-secret-name

Can you please leave a note that the node has to go through the power lifecycle before removing the taint or else we are going to have data inconsistency/corruption problem

Added as Important note above.

@Madhu-1, are we good with this solution to address the older PVCs?

Yes we are good with this one 👍🏻

we dont need restart here, just mention that power cycle( includes shutdown and start it )

full shutdown and restart cycle to full power cycle