Repository navigation
bug(storage): a replicated volume can end up with no filesystem, and the mount retry never recovers #3854
Description
A replicated PVC can bind, attach and reach UpToDate on all three nodes while carrying no filesystem. Once that happens nothing recovers it: kubelet retries NodePublishVolume, CSI retries the mount, the mount keeps failing on the missing superblock, and the pod stays in ContainerCreating until something above it times out.
Seen in 31873151114, where it took out the install phase. Found while reading artifacts for #3851; that issue tracks the install-phase flake as a whole, this is the one piece under it that is a defect on its own.
What the artifacts show
The volume is healthy at every layer below the filesystem. linstor r l gives pvc-047048c2-333a-484f-96c1-56f870b59f65 as UpToDate with Conns: Ok on srv1, srv2 and srv3, and dmesg shows the resync completing as Began resync as SyncSource (will sync 0 KB [0 bits set]).
The mount fails anyway:
NodePublishVolume failed for pvc-047048c2-333a-484f-96c1-56f870b59f65: mount failed: exit status 32
Mounting arguments: -w-w -t ext4 -o _netdev /dev/drbd1001 /var/lib/kubelet/pods/.../mount
Output: mount: ...: wrong fs type, bad option, bad superblock on /dev/drbd1001, missing codepage or helper program, or other error.
96 hits for that volume in the csi-node log, retried from 09:06:39 through 09:24:16 and past the end of the capture. Every retry is the same mount of the same empty device. The sibling volume /dev/drbd1000 on the same node mounted fine at 09:06:28, so this is per-volume.
Where it probably comes from
The only LINSTOR error report in the entire run lands one second before srv3 attaches this volume:
6A802B26-E4EA0-000000 2026-08-15 09:06:17 S|srv3 StorageException: Device did not get ready within 2000ms
That is a hard 2s wait for the device node to appear. My reading is that it aborted volume preparation before the filesystem was created, and DRBD then replicated the empty device faithfully. This is ordering plus the single error report, not proof: I have no log line that says mkfs was skipped, and the csi-node log does not print the format step.
Why it is worth a separate issue
Two things, independent of whether the 2s timeout is really the trigger.
The retry loop cannot fix this failure. CSI retries the mount, and the mount is not the step that failed. A volume that reaches this state stays broken for as long as the workload lives.
The 2s device wait is not tunable from anything Cozystack sets, and on a busy host it is easy to miss. If that is the trigger, the same window is reachable on real clusters, not only in e2e.
What would confirm it
A run that hits it again with mkfs visible: either raise the csi-node log level so the format step is logged, or check the device directly at failure time with blkid /dev/drbdXXXX and linstor r l --faulty. If blkid comes back empty the chain is confirmed, and the fix belongs upstream in linstor-csi: detect the missing filesystem in NodeStageVolume and format instead of retrying the mount forever.
Activity
lexfrei commentedon Aug 20, 2026
The mechanism is now measured end to end on two instances (runs 31873151114 and 32350714896, both on srv3), and it is harder than the event-order reading recorded earlier: the volume is declared ready without a filesystem ever having been created, and no retry anywhere in the stack can repair that.
The chain, from the second run's artifacts. The satellite throws
StorageException: Device did not get ready within 2000msinDrbdLayer.waitForValidStateForPrimary(the error report's own stack trace, LINSTOR 1.33.3). That moment is the volume's only chance to get a filesystem: mkfs is the satellite's job, driven byFileSystem/Type = ext4on the volume definition, and writing it requires the DRBD Primary role. The node's kernel log shows what the exception means physically: the seven earlier attach cycles for this volume each carry arole( Secondary -> Primary )transition about 180ms after attach, and the failing cycle carries none at all, attach at 09:33:21.896, detach at 09:33:22.644. LINSTOR then proceeds as if nothing happened: the retry task is removed, the node is notified as initialized, replicas land on srv1 and srv2, resync transfers 0 KB because there is nothing to sync,Linstor/Drbd/InitialUptoDateOnis set, CSI reports success and the PVC binds. NodePublish then fails 44 times withwrong fs type, bad option, bad superblock on /dev/drbd1000, and it cannot ever succeed: CSI repeats the mount, never mkfs, and all csi-node logs of the run contain zero mkfs lines, healthy volumes included.The first instance pins the same shape with a clean counter: one resource-definition creation, the same exception three seconds later, and the promotion count for that volume is zero on srv3 against 48 on srv2, one per mount attempt.
One connection refuted along the way: on the second run this hit during the 8th autoplace cycle of the #3851 mechanism, which looked like a link, but the first instance had no autoplace cycle at all, so the co-occurrence is coincidence, not a precondition.
The root sits upstream: the 2000ms budget is a fixed constant with no knob, the initialization failure is neither fatal to volume creation nor retried, and the result is a bound PVC over an empty device. Filing this with both error reports upstream is the next step. A local safety net is possible, since the defect has an exact signature (bound PVC, all replicas UpToDate, a 0-bit resync and NodePublish failing on wrong fs type), but it would carry permanent cost and is not proposed while upstream has not answered.
lexfrei commentedon Aug 20, 2026
Source-level follow-up: the three claims above are now anchored in the upstream tree rather than inferred from behaviour.
The 2000ms budget is
private static final long HAS_VALID_STATE_FOR_PRIMARY_TIMEOUT = 2000;atDrbdLayer.java:100in v1.33.3, with no property or config behind it. The unrecoverability is an ordering:runPostInitializationIfNeeded(DrbdLayer.java:948-990) first callssendNotifyCtrlVlmNrsWereInitialized(...)and only then reaches theMkfsUtils.needsToCreateFs(...)branch whosewaitForValidStateForPrimarythrows, so by the time the exception fires the controller has already been told the volume is initialized; the controller setsVolumeDefinition.Flags.DRBD_INITIALIZEDand commits (VlmDfnInternalCallHandler.java:144). Every later device-manager pass then entershasLocalStltWonUpToDateRace(DrbdLayer.java:1572-1583), which carries the comment "if DRBD_INITIALIZED is set, the race is over" and returns false, so the whole initialization block, mkfs included, is skipped for the rest of the volume's life. Upstream knows the placement is awkward: the mkfs call sits next to a// TODO we should move the mkfs outside of the devMgr, but that is a different issue/MR. Both the constant and the ordering are unchanged on current master (CHANGELOG at 1.35.0-rc.1, 2026-08-13), so upgrading does not help. An upstream report with the two error reports and these anchors is being prepared.lexfrei commentedon Aug 20, 2026
Filed upstream as LINBIT/linstor-server#526, with both error reports, the promotion-counter marker and the source anchors verified at the v1.33.3 tag.