bug(tests): e2e install phase flakes under parallel load, always surfacing as a monitoring-system timeout on VLCluster

Status (2026-08-16, supersedes the framing below where they disagree)

Eight install-phase failures over 2026-08-14 to 2026-08-16 have been read from their artifacts (details in the comments). What holds: every failure is the same pair of replicated PVCs, vlstorage-db-vlstorage-generic-0 and -1, the only two replicated volumes the install creates, provisioned in parallel with autoPlace: 3 on three nodes. Seven of eight are one mechanism, A: linstor-csi places replicas incrementally, the first call puts one replica on a node, the second call asks for the remaining two and is refused with Not enough available nodes in under a millisecond from the controller's in-memory state, no satellite involved, no error report, and the refusal repeats for 12 to 13 cycles over 14 to 19 minutes. In every run the two volumes take their first replicas within 0.6 s of each other and never on the same node, then one of them completes and the other stays stuck; which one loses does not depend on which started first. The eighth failure is mechanism B, a volume that binds UpToDate with no filesystem on it, filed separately as #3854.

What is ruled out with carriers: satellite capacity reports (volumes land on all three nodes in the same second), free space (the failing runs had more free space on their tightest node than any green run), registration racing (50 to 74 s of margin), tiebreaker or diskless replicas, a neighbour's replica on the node, placement constraints (none configured), image pulls, etcd bootstrap and the grafana ClusterIP errors (co-symptoms of the same window, both self-healing), host load (reproduces with zero neighbouring jobs), and a slow sandbox (the failing install runs faster than the green control up to the step that waits on the non-converging object, so a bigger timeout changes nothing).

What is open: the reason the autoplacer rejects the two idle nodes on the second call, and why the rejection sticks after the winner is fully placed. The artifacts cannot carry it because an autoplace refusal writes no error report and the controller logs only the aggregate. The leading candidate is that the second call, carrying Do not place with resource: <itself> and Replica count: 3, drops the node holding the first replica and then checks the remaining two candidates against a demand of three; a source-level check against the pinned LINSTOR and linstor-csi is in progress. Discriminator for the next live occurrence: linstor storage-pool list, linstor node list and a manual linstor resource create <rd> --node <idle-node> --storage-pool data inside the refusal window, which returns the per-node reason instead of the aggregate.

What happens

The e2e install phase intermittently fails before any suite runs: make install-cozystack exits 1 with Install failed (no retry, see diagnostics below) after the install budget is spent waiting on HelmReleases that never leave InProgress. The suites never start, so the run produces no chainsaw-report, and the failure is visible only in the install-phase diagnostics.

Three occurrences in the last two days, all with the same visible trunk:

date (UTC)	run	visible timeout	co-symptoms in the same log
2026-08-15 18:06	31827081398	`monitoring-system` waiting for `VLCluster/tenant-root/generic`	etcd bootstrap `discovery failed: cannot fetch cluster info from peer urls` (EOF from peer); LINSTOR `0/3 nodes... failed to enough replicas on requisite nodes: Not enough available nodes` for the `vlstorage` PVC; grafana `dial tcp <clusterip>:5432: connect: no route to host`
2026-08-15 07:54	31873151114	same: `monitoring-system` on `VLCluster/tenant-root/generic`	etcd `discovery failed` twice at 09:06; grafana `no route to host` to postgres
2026-08-16 10:38	31939731000	same: `monitoring-system` on `VLCluster/tenant-root/generic`	etcd releases succeeded on this run; cascade otherwise identical

The cascade is identical each time: VLCluster/tenant-root/generic stays InProgress, so monitoring-system hits its HelmRelease timeout, so monitoring hits its timeout waiting on monitoring-system, so the tenant-root upgrade times out waiting on monitoring, and the install budget expires.

What sits underneath, and what is not yet proven

The visible timeout names the top of a dependency chain, not a cause. Two of the three runs carry storage-shaped failures underneath: the vlstorage PVC that VLCluster waits on is LINSTOR-replicated, and one run shows LINSTOR unable to place replicas on any node, while two runs show the tenant etcd pods dying at bootstrap unable to reach their peers. One hypothesis that fits all three is that replicated storage is slow or failing to provision during the busy install window, and everything above it (VLCluster, etcd data, postgres behind the grafana no route to host) is queuing on that. This is a hypothesis by adjacency, not a mechanism: nobody has yet read the LINSTOR resource states, the PVC event streams and the etcd pod logs from the captured artifacts side by side.

Load correlation, unproven but consistent

All three occurrences happened while six to seven PR e2e workflows ran in parallel. Installs on a quiet host pass: a manually dispatched full-suite run on the same code base (31840272448) installed cleanly and went green 21/21, and single-PR runs between the bursts install fine. The class also predates this week: install-phase failures with etcd leader churn during the install window were observed earlier in August, so this is a standing failure mode, roughly 3 out of 20 runs over the last two days, second in weight after the node-join flake of #3513.

What the recent fixes do not touch

The worker sizing and diagnostics work for #3513 (#3839, #3840, #3836, #3841, #3843) all concern the nested tenant workers inside the kubernetes suites. This class fails earlier, on the sandbox cluster itself, before any tenant cluster exists, so none of that work addresses it.

Artifacts available for investigation

Each failing run uploaded a cozyreport artifact captured at the moment of failure: cluster-wide events, HelmRelease statuses, pod listings, crust-gather snapshots of the failing namespaces, and the install-phase log itself. The three runs above are within the artifact retention window now. A mechanism-level readout should be possible without touching a cluster: LINSTOR resource and node states at failure time, the vlstorage PVC event stream, tenant etcd pod logs around the discovery failed window, and timing of image pulls against the install budget are all in there.

What a fix needs

A mechanism first. The candidate levers differ sharply depending on what the artifacts show: LINSTOR placement failing under load points at storage readiness gating or replica placement; etcd bootstrap churn points at the tenant etcd lifecycle under slow disk; a pure budget shortfall under cold caches points at install-phase budgets. Bumping HelmRelease timeouts blind is the one move that is known to be wrong here, since the failing runs spend their budget waiting on something that is not converging, not converging slowly.

Fourth occurrence, same trunk: run 31939731000 (2026-08-16 09:41 UTC, branch fix/cozytest-negated-assertions) failed Install Cozystack into sandbox with monitoring-system timing out on VLCluster/tenant-root/generic, then monitoring, then the tenant-root upgrade, in that order. No etcd discovery failed on this one, so the etcd symptom is not required for the cascade to happen; the etcd releases in that log show Helm install succeeded. This run was one of six parallel e2e workflows started within a minute of each other. Count on this class is now 4 of roughly 26 e2e runs over three days.

Correction to the table above, and the first mechanism-level readout.

The first row is misdated: run 31827081398 was created 2026-08-14 18:06 UTC, not 15.08, and it carries two cozyreport artifacts because the e2e job was re-run: the install-phase capture from the first attempt (job 18:24 to 19:01 UTC, artifact 9230781794) and a later one from the second attempt that reached the suites. gh run download -n cozyreport fetches the later one, so anyone reading this failure needs gh api repos/cozystack/cozystack/actions/artifacts/9230781794/zip.

From that first-attempt capture, the chain under the visible timeout is fully resolved. PVC tenant-root/vlstorage-db-vlstorage-generic-0 (StorageClass replicated) stays Pending for 19 minutes, so vlstorage-generic-0 cannot schedule (pod has unbound immediate PersistentVolumeClaims), so VLCluster generic never becomes Ready, so monitoring-system hits its 15-minute install timeout, and the rest is the cascade. Its sibling PVC vlstorage-db-vlstorage-generic-1 provisioned normally with three UpToDate DRBD replicas.

The failure is narrow, not a LINSTOR outage: at capture time all three satellites are Online, the data ZFS pool is Ok on all three with 158, 119 and 127 GiB free, no error reports, every existing resource UpToDate. The CSI log carries the actual refusal: CreateVolume ... ResourceExhausted: Not enough available nodes with Replica count: 3 / Additional replica count: 2 / Node names: srv1 srv2 srv3 / Storage pool name: data / Layer stack: DRBD STORAGE, against StorageClass parameters autoPlace=3, layerList=drbd storage, storagePool=data, allowRemoteVolumeAccess=true, 10 GiB requested. The controller log shows the same cycle twelve times between 18:41 and 18:55: resource definition created, one replica placed on srv3 every time, error, resource deleted, retry. srv1 and srv2 never received a replica; their dmesg has no line for this resource.

Registration racing is ruled out from the same log: the controller came up at 18:39:13, all three satellites were ONLINE by 18:39:55 with pools created at 18:39:57, and the first CreateVolume arrived at 18:41:05.

One anomaly sits in the window without a proven link: the LINSTOR satellite DeviceManager on srv1 held a single cycle from 18:42:06 to 18:56:15 with no operation inside it, and srv2 the same from 18:43:22 to 18:56:24, while srv3 cycled normally. The stall covers the failure window, but the first refusal at 18:41:19 predates it, so this is adjacency for now. Sandbox host at 19:00: three QEMU VMs at -smp 8 -m 24576, lifetime CPU 433, 353 and 416 percent, memory 128 GB with 52 GB available.

What does not add up yet: every criterion named in the error text (pool present, space, node online) holds at capture time, so either the autoplacer filters on something the templated message does not name, or the state at refusal time differed from the snapshot. The other two runs and a green control are being read the same way next.

Second readout, covering all three captured runs and a green control. The picture is two mechanisms under one object, and three of the co-symptoms listed in the issue body turn out not to be in the chain at all.

The common object

Every one of the three failures is the same PVC: tenant-root/vlstorage-db-vlstorage-generic-0 on StorageClass replicated, the volume of vlstorage-generic-0. Its sibling -1 provisions and binds normally in every run. Everything above it (VLCluster not Ready, monitoring-system timing out, monitoring, tenant-root) is the same cascade each time.

Mechanism A: LINSTOR autoplace refuses two of three nodes (runs 31827081398 attempt 1, 31939731000)

CreateVolume fails with ResourceExhausted: Not enough available nodes. In both runs LINSTOR places exactly one replica, always on srv3 (24 controller lines on 'srv3' in the second run, none on srv1 or srv2), then rolls back; csi-provisioner retries twelve times with backoff over about fourteen minutes. The resource group from the LINSTOR DB dump has replica_count: 3, pool_name: ["data"], layer_stack: ["DRBD","STORAGE"] and empty node_name_list, replicas_on_same, replicas_on_different, do_not_place_with_rsc_list, so no placement constraint is configured.

What does not close: every criterion the error text names holds at refusal time. All three satellites Online, pool data present on all three with 158, 119 and 127 GiB free against 10 GiB requested, and in the first run the linstor sp l snapshot was taken twelve seconds after the last refusal, so this is not stale data. The autoplacer is filtering on something the templated message does not name, or on state the bundle does not carry. Autoplace refusals create no LINSTOR error report (the error-report index is empty in both runs) and the controller logs only the aggregate Not enough available nodes without naming which node failed which check. That is an instrumentation gap in its own right. The discriminator for the next occurrence: at refusal time, linstor storage-pool list, linstor node list, and a manual linstor resource create <rd> --node srv1 --storage-pool data, which returns the real per-node reason instead of the aggregate.

The long DeviceManager cycles on srv1 and srv2 mentioned above are withdrawn as a lead: Begin cycle N+1 follows End cycle N within milliseconds, so a long cycle with no operations inside is the satellite idling because it was given no work, a consequence rather than a cause.

Mechanism B: volume bound, but no filesystem on it (run 31873151114)

The PVC binds, the DRBD resource pvc-047048c2 is UpToDate on all three nodes with Conns: Ok, and mount -t ext4 /dev/drbd1001 fails forever with wrong fs type, bad option, bad superblock (96 mentions in the csi-node log from 09:06:39 onward). The single LINSTOR error report of the whole run is StorageException: Device did not get ready within 2000ms on srv3 at 09:06:17, one second before srv3 started attaching this volume. The reading that fits: the two-second device-ready timeout aborted volume preparation before mkfs, DRBD then faithfully replicated an empty device, and CSI retries mount rather than mkfs, so it can never converge. The link from the StorageException to the skipped mkfs is by timing and adjacency, there is no explicit "mkfs skipped" line, so treat it as strongly supported rather than proven.

Co-symptoms that are not in the chain

etcd discovery failed: present in all three runs (the body above says the third run had none, that is wrong: 10:18:53 and 10:19:24), and in all three the etcd pods are Running with restartCount: 1 at capture time, so bootstrap converged on the second attempt. It coincides with the first LINSTOR refusal to within seconds because both start when the platform rolls out tenant-root, not because one causes the other.

grafana no route to host to the postgres ClusterIP: in each run the client hits the ClusterIP a few seconds after the single backend pod started (18:57:31 vs pod start ~18:57:26; 09:22:41 vs ~09:22:32; 10:35:07 vs ~10:35:11), well before readiness, and grafana ends Running 1/1 after two or three restarts. This is the known ClusterIP-with-no-ready-endpoints shape under the kube-proxy replacement. The postgres PVCs are on the local StorageClass and are all Bound, so they are outside the LINSTOR failure.

Cold image cache: the longest pull in any of the three runs is 21.9 to 30.7 seconds (alerta/alerta-web:9.1.0), outside the vlstorage window; 234 to 243 successful pulls per run, zero ImagePullBackOff, zero ErrImagePull.

Never, not slowly

Green control run 31840272448: Install Cozystack 11m55s, Configure Tenant and wait for applications 5m32s of a 20-minute budget. Failing run 31939731000: Install Cozystack 10m48s, a minute faster than the green one, and Configure Tenant runs to its 20m03s ceiling with timed out waiting for the condition on helmreleases/monitoring. The sandbox was not slower; the step waits on an object that does not converge. The budget lives at hack/e2e-install-cozystack.bats:236 (kubectl wait ... --timeout=20m); raising it does nothing here, which is now a measured 5m32s-versus-never rather than a general remark.

Load correlation withdrawn. The issue body says all three failures happened under six to seven parallel workflows; measured against the actual run windows (every Pull Request and Nightly run overlapping the vlstorage refusal window, by createdAt/updatedAt), the numbers are 3, 9 and 1 concurrent e2e runs for the three failures, and 3 for the green control's install phase. Mechanism A reproduced with a single neighbour on the host. So parallel e2e load on the runner host is not a precondition for this class, and the "loaded host" framing above should be read as unproven adjacency, not as a condition. What the three failures share is the object (vlstorage-db-vlstorage-generic-0 on replicated) and the timing inside the install (the first tenant-root rollout), not the host's neighbours.

Two closing points from the same investigation, then the scope is wider than the three runs above.

On concurrency, one precision so the withdrawal above is not overread: counted as e2e jobs whose runtime overlaps the refusal window (rather than workflow runs, whose build stages precede e2e by about 25 minutes), the neighbours are 1 for the 2026-08-14 case, 9 for the 2026-08-15 case, and 0 for the 2026-08-16 case, where the only workflow alive on the host started its e2e job at 10:46 UTC, after the vlstorage refusal window had closed. Both mechanisms therefore reproduce with the host otherwise idle, and each job runs on its own ephemeral runner VM in any case, so neighbouring jobs never share the sandbox. Load is not part of this class.

On scope: over 2026-08-14 to 2026-08-16 the install step went red in 8 e2e jobs, not 3. Beyond the three read above: runs 31880859267, 31880862372, 31884498621, 31895434081, 31926679135. That is 8 of 97 jobs in which the step reached a verdict, and all five extra artifacts are still within retention, so the A-to-B ratio can be measured on a real denominator instead of a sample of three.

Practical trap for whoever reads these next: run 31827081398 carries two cozyreport artifacts because its e2e job was re-run, and gh run download -n cozyreport returns the later one, which does not cover the install failure; the install-phase capture is artifact 9230781794.

Mechanism B is filed on its own as #3854, since a replicated volume that binds with no filesystem and a mount retry that can never repair it is a storage defect regardless of what triggers it, and the trigger candidate (a hard 2s device-ready wait in the satellite) is reachable outside e2e. Mechanism A stays here: the missing piece is per-node attribution of the autoplace refusal, which no captured artifact carries, and the discriminator for the next occurrence is spelled out above.

All five remaining install failures read. The class is now measured on the full denominator instead of a sample, and two things I wrote earlier in this thread turn out to be artifacts of reading only three runs.

Ratio

Eight install-phase failures over 2026-08-14 to 2026-08-16. Seven are mechanism A (LINSTOR autoplace refuses two of three nodes), one is mechanism B (volume binds with no filesystem, now #3854), none is a third mechanism. The five read here are 31880859267, 31880862372, 31884498621, 31895434081, 31926679135, all mechanism A, all with the same ResourceExhausted: Not enough available nodes on a replicated PVC of the vlstorage-generic StatefulSet. None of them carries a wrong fs type mount loop, and the LINSTOR error report index is empty in all five, same as in the two A runs read earlier.

None of the five was a re-run, so gh run download -n cozyreport returns the right artifact for each. The two-artifact trap applies only to 31827081398.

Two corrections

The failing object is not always ordinal 0. Across the five: vlstorage-db-vlstorage-generic-1 in 31880859267, 31880862372 and 31926679135, -0 in 31884498621 and 31895434081. What is fixed is the pair, not which member of it fails.

The surviving replica is not always on srv3. It is on srv1 in three of the five and on srv3 in two. Counting the seven A runs, srv3 four times and srv1 three times, and never srv2.

What the install actually asks of LINSTOR

The whole install creates exactly two replicated PVCs, both from vlstorage-generic, and nineteen local ones. The replicated StorageClass is autoPlace: 3 with layerList: drbd storage on a three node cluster, and volumeBindingMode: Immediate, so both PVCs are provisioned in parallel rather than serialised behind pod scheduling. local is a separate resource group with replica_count: 1 and layer stack STORAGE only.

In every one of the five runs the two volumes take their first replica within 0.6 s of each other, and never on the same node. Then one of them goes on to take the remaining two nodes and the other is refused, retried, rolled back and refused again for the rest of the capture. Which one wins is not decided by who started first: in 31884498621, 31895434081 and 31926679135 the volume that ends up stuck placed its first replica before the winner did.

srv2 is never picked first. Across the five runs there are ten first-placement events, zero of them on srv2.

The refusal is not a capacity report race

The open hypothesis was that the first CreateVolume after controller startup outruns the free-space reports from two of the satellites. It does not hold, and three separate measurements close it.

Timing first. In all five runs the controller acquires its lease, all three satellites reach ONLINE, and all three data pools are registered well before any volume is requested. The gap from the last pool registration to the first CreateVolume is 50, 65, 65, 70 and 74 seconds. There is no window to race in.

Second, concurrent positive controls. In the same seconds as the refusal, LINSTOR successfully places volumes on the very nodes it is refusing. In 31880859267 the failing volume is refused at 11:33:26.597 while local volumes land on srv2 at 11:33:24.002 and on srv3 at 11:33:24.707, and the sibling replicated volume takes srv3 at 11:33:24.026, srv2 at 11:33:27.535 and srv1 at 11:33:28.367. A controller with no free-space data for srv2 and srv3 could not have placed anything there. The same shape holds in the other four runs: every one has placements on all three nodes inside the window between the failing volume's resource definition being created and its first refusal.

Third, the refusal is too fast to include a free-space query. linstor-csi issues two AutoPlaceRsc calls per attempt. The first places one replica. The second asks for the remaining two and fails immediately: across the 62 refusals in these five runs the gap from the AutoPlaceRsc REST call to the Not enough available nodes error is 0 to 43 ms, with the fastest in every run under a millisecond, for instance AutoPlaceRsc at 11:47:15.486 and the error at 11:47:15.487. A sub-millisecond decision cannot contain a round trip to three satellites, so at least those refusals come out of state the controller already holds.

Free space is not the constraint either, and this is now measurable against green runs rather than against a snapshot of the same run. Four green runs finish with 6 DRBD replicas placed and a tightest-node free capacity of 97.32, 101.38, 107.47 and 109.51 GiB. The five failing runs sit at 111.54 to 119.67 GiB on their tightest node, that is, every failing run had more free space everywhere than every green run, while refusing a 10 GiB request.

What the refusal is not, either

Worth recording so nobody re-checks these. No placement constraint is configured: resource group sc-6554bcd0 has replica_count: 3, pool_name: ["data"], layer_stack: ["DRBD","STORAGE"] and empty node_name_list, replicas_on_same, replicas_on_different and do_not_place_with_rsc_list. The nodes carry no restricting properties, only Aux/topology/* and NodeUname. No diskless resource and no quorum tiebreaker is ever created for the failing volume, so exclusion by "the node already holds this resource" cannot be the reason for two nodes at once. linstor-affinity-controller sees the resource and logs no matching PV and no saved PV configuration, nothing to do. Only the satellite that received the single replica ever mentions the volume, on the other two there is no line for it in either the satellite log or dmesg.

The obvious reading, that a node is excluded because it already carries another DRBD resource, is also ruled out. In 31880859267 the winning volume's autoplace takes srv1 at 11:33:28.367, and srv1 was at that moment still carrying the losing volume's replica, which only starts being deleted at 11:33:28.576. Holding a replica of the other volume did not disqualify a node in that direction.

And the refusal is not transient. It persists for 14 to 19 minutes across 12 or 13 cycles per run, long after the winning volume has all three replicas and the placement burst is over. The last refusal in 31880859267 is at 11:52:42, about a minute before the linstor sp l snapshot that shows 156, 117 and 131 GiB free on the three nodes.

Where this leaves it

The remaining gap is unchanged in kind but narrower in scope. It is no longer "LINSTOR cannot find nodes"; it is "the second, incremental autoplace call refuses two named nodes in one millisecond, while a sibling volume with identical parameters succeeds on those same nodes seconds earlier and later". Autoplace refusals produce no error report, and the controller logs only the aggregate without naming which node failed which check, so the reason is not in the bundle. That instrumentation gap is the thing blocking a root cause.

The discriminator for the next occurrence stands, with one addition. At refusal time, capture linstor storage-pool list, linstor node list, linstor resource list, and then run linstor resource create <rd> --node <refused-node> --storage-pool data by hand, which returns the per-node reason instead of the aggregate. Raising the LINSTOR controller log level for the autoplacer would give the same answer without the manual step, if that is reachable through the piraeus operator.

Separately, and independent of the root cause: autoPlace: 3 on exactly three nodes leaves no spare. Any single node being filtered out for any reason turns provisioning into a permanent failure rather than a degraded placement, and the retry loop cannot recover it because every cycle starts from the same state.

One candidate for the sticky part that the artifacts cannot settle but the source can. Every refused second call carries Replica count: 3 / Additional replica count: 2 / Do not place with resource: <the same PVC>. If the autoplacer in the pinned LINSTOR applies that exclusion by dropping the node that already holds the first replica and then checks the candidate set against the full replica count of 3 rather than the additional 2, the request has exactly two eligible nodes against a demand of three and is refused instantly from in-memory state, with every pool healthy, which is precisely the shape observed: sub-millisecond, no satellite involved, no error report, and it stays refused for as long as the first replica sits on its node. It would also explain why the winner of the pair never hits it: its second and third calls each ask for one more replica at a time and never for two at once. This is a hypothesis with a source-level check, not a measured mechanism. The check is a side-by-side of the second AutoPlaceRsc request of the loser and of the winner in the same run (the request bodies are in the controller REST log), and a read of the autoplacer's handling of do_not_place_with_rsc against the requested count in the LINSTOR version pinned by the chart. If it holds, the fix is in linstor-csi's incremental placement (ask for the remaining count, or place all three in one call), and it is upstream.

The "three demanded against two eligible" hypothesis does not hold. I read it against the pinned sources and the arithmetic it assumes is not there.

The pins are upstream linstor-server 1.33.3 and linstor-csi v1.11.2 (packages/system/linstor/Makefile; the v1.6.0 in the image tag is our version, not upstream's). None of our patches touch placement: the server patches sit in the toggle-disk, LUKS sizing and DrbdAdm paths, the CSI patches in Clone, VolFromSnap and Attach.

Placement is incremental, that part was right. The default policy is AutoPlaceTopology and the StorageClass does not override it, so Scheduler.Create places one replica on a preferred node first (PlaceCount: 1, AdditionalPlaceCount: 1, one node in the list) and then autoplaces the rest. The error we capture, failed to enough replicas on requisite nodes, exists at exactly one place in the CSI, inside that second step, so the failing call is the one that asks for the remainder.

What that second call sends is narrower than we assumed. It carries a node name list and nothing else. It does not send an additional place count and it does not send a do-not-place-with list. Both of those appear in the error text because the controller adds them: it appends the resource's own name to do-not-place-with, and it computes the additional count itself.

And it computes it correctly. additional = replica_count - already_deployed, so 3 - 1 = 2, which is precisely the Additional replica count: 2 we see in the error text. The selector then computes its own demand the same way and compares the candidate set against that 2, not against the 3. The Replica count: 3 line above it is just the merged filter echoing its own field; the demand is the line below. So the node holding the first replica is indeed dropped, exactly as the hypothesis said, but the demand drops with it. Two eligible nodes against a demand of two is a selection that succeeds.

Two of the four sub-claims do hold, and they close two open questions. The refusal is decided entirely from controller memory: the autoplacer filters, rates and selects over the controller's own storage pool map, checking only whether the peer is online and reading a cached free-space estimate. There is no satellite round trip on that path, so sub-millisecond is the expected latency and not evidence of anything being skipped. And the response is built with setSkipErrorReport(true), so the empty error report index is deliberate rather than a gap in what we captured.

Which leaves the refusal needing something else. Demand two, one node accounted for by the self-exclusion, so at least one more node has to be dropped by the filter. Every other filter criterion is resource-independent (online, free space, layers, storage pool name, provider kind, node properties), and we already measured those healthy by watching other volumes land on the same nodes in the same second. The only exclusion specific to this resource definition is the do-not-place-with one, and there is an asymmetry in it worth checking: the demand computation ignores resources flagged DELETE, EVICTED or EVACUATE, while the node filter ignores only DELETE. A leftover replica of the same resource in one of those other states would be invisible to the demand and visible to the filter, leaving one candidate against a demand of two, instantly, with no error report, and sticky until it is cleaned up. That is a shape that fits, not a mechanism I have measured.

Cheapest next step, and it may already be in the bundles we have. The failure text prints a Node name line whenever the node name list is non-empty, and this call always carries one, so the full details block of the Not enough available nodes response records which nodes the CSI actually offered. If it names all three, the drop happened inside the filter. If it names fewer, the requisite list was short and the question moves to the topology side. After that it needs a live occurrence: linstor resource list --all with the flags column at refusal time, and the controller at TRACE, where Autoplacer.Filter: Disqualifying node '<name>' as ... gives both the node and the reason.

On upstream: no fix. Nothing in the linstor-server changelog through 1.35.0-rc.1 touches autoplace counting, do-not-place-with, or this refusal. linstor-csi 1.11.3, the release right after our pin, does fix concurrent CreateVolume calls sharing a resource group, but for spurious "already exists" errors, which is not our error text. Worth a bump on its own merits, not as a fix for this.

The open branch above is settled from the artifacts already captured. The full DETAILS block of the refusal (csi-controller log, run 31939731000) carries the node list the second call offered, and it is all three: Node names: srv1 srv2 srv3, alongside Additional replica count: 2 and Do not place with resource: <itself>. So the short-requisite branch is dead: csi asked for two more replicas across all three nodes, the self-exclusion removes the one node holding the first replica, two eligible nodes remain against a demand of two, and the placer still refuses. That means the filter disqualifies at least one more node for a reason that is resource-dependent (the concurrent positive control shows both other nodes accepting other volumes in the same second), and the only such path visible in the pinned source is the flag asymmetry named above: a leftover replica of the same resource carrying a flag the demand calculation ignores but the node filter does not. The rollback loop this failure runs in (create one replica, fail, delete, retry, twelve times) is exactly the kind of churn that leaves such leftovers. Next live occurrence: linstor resource list --all inside the refusal window, flags column included, plus the controller on TRACE where Autoplacer.Filter: Disqualifying node '<name>' as ... names the node and the reason.

Two more instances: runs 32025422447 (2026-08-17) and 32117285553 (2026-08-18, a manually dispatched nightly on main). Both fail the install with the monitoring HelmRelease timing out on VLCluster tenant-root/generic staying InProgress, the vlstorage PVCs Pending, and LINSTOR answering ResourceExhausted with not enough available nodes. Checked for the cilium address collisions that explained the other install failures from the same symptom pool: none present in either snapshot, so this cause stands alone here. The second run also carries the new runner identity record (VM.Standard.E4.Flex, us-ashburn-1), the first instance of this flake with known hardware.

Detail on the two instances above, because the refusal message turns out to carry both counts the open question is about, visible in the artifact without live access:

Auto-place configuration details:
   Replica count: 3
   Additional replica count: 2
   Node names: srv1 srv2 srv3
   Storage pool name: data
   Do not place with resource: pvc-<the resource itself>

Additional replica count: 2 is the demand after the first replica is placed, so the second call asks for two while exactly two nodes are idle and online, each with four orders of magnitude more free space than the 10485760 bytes the criteria require, and the refusal still happens. The printed Replica count: 3 next to it is a configuration echo rather than proof of what the check compared against, but it is consistent with the leading candidate in the status section.

Both runs match mechanism A on every recorded marker: 12 ProvisioningFailed cycles, a 14-minute refusal window (14m11s and 14m12s), all three satellites Online, the error report index empty, and the winning volume of the pair at three UpToDate replicas. The losing member differs between the runs (generic-1 in the first, generic-0 in the second), one more sample for the note that the loser does not follow start order.

One more instance, with an unusual property: run 32157692709 (2026-08-18) replayed a July 24 main tree (a80f134) as an environment probe for a separate investigation, and hit this flake. Signature matches the registry: vlstorage-generic-1 PVC stuck Pending with the LINSTOR autoplace error Not enough available nodes, then the monitoring HelmRelease times out waiting on monitoring-system. A month-old tree reproducing this today says the trigger lives in the environment or in a long-standing race, not in anything recent on main.

Second instance today, and the first one carrying a runner identity record: run 32138538112 (PR lane, current main plus diagnostics) failed with the same signature, vlstorage-generic-0 PVC Pending on the LINSTOR autoplace error Not enough available nodes, monitoring HelmRelease timing out behind it. The new sandbox-host identity capture says this run landed on VM.Standard.E4.Flex (AMD EPYC 7J13, Milan), us-ashburn-1 (the capacity fallback region for this pool), FAULT-DOMAIN-3, 16 OCPU. One data point says nothing about causation yet, but from now on every instance of this flake comes with its hardware fingerprint, so a shape or region bias will become checkable once a few more land.

Today's two instances turned out to be two different mechanisms, so first the split: run 32350576394 is this issue's mechanism (autoplace refusal), while run 32350714896 is the volume-without-filesystem chain tracked in #3854 and does not belong here. Everything below is about the first run.

The autoplace mechanism is now confirmed on live controller output, with one long-standing hypothesis closed. All 12 CreateVolume cycles for vlstorage-db-vlstorage-generic-1 print the same DETAILS block: Additional replica count: 2, Node names: srv1 srv2 srv3, and a do-not-place-with entry that excludes only srv1, where the first replica already sits. So the demand is two, the candidate set after the exclusion is exactly two, and one of those two candidates is rejected for a reason the controller never writes down: Assembled error reports; count 0 throughout, only the aggregate Not enough available nodes reaches the log, which is the controller's documented skipErrorReport behaviour rather than a capture gap. The earlier hypothesis that the placer compares against the full replica_count: 3 is refuted by these numbers.

What was also ruled out on the same run: the resource group carries no constraints at all, the storage pools have 109 to 158 GiB free against the 10 requested, all three nodes are Online, and other replicated volumes placed successfully on srv2 and srv3 inside the failure window, so the rejection is specific to this resource, not to a node. The first failure lands in the same second as the sibling volume's placement step taking srv1 and srv2, which plausibly explains the onset, but the next 11 failures repeat for 14 minutes after the sibling fully settled, so the race does not explain the stickiness. The sticky discriminator is still the open question.

Two follow-up thoughts, one instrument and one design note. The instrument: the sandbox is destroyed right after this failure path, so the diagnostics can afford an active probe there, and linstor resource create <rd> --node <free-node> --storage-pool data in that window would return the real per-node error the aggregate hides. The design note, stated as exposure rather than cause: autoPlace: 3 on a three-node sandbox leaves zero spare candidates, so any single-node rejection wedges provisioning permanently; lowering it to 2 in the e2e StorageClass would remove the exposure at the cost of no longer exercising the three-replica path.

Another instance of the autoplace mechanism, run 33645509783 (PR lane). Same shape: vlstorage-db-vlstorage-generic-1 Pending for 19 minutes, 12 CreateVolume cycles between 15:46:54Z and 16:01:09Z, in every cycle the first replica landed on srv1, the placer then refused with Not enough available nodes, and the srv1 replica was rolled back. The DETAILS block matches the 2026-08-20 one: Additional replica count: 2, Node names: srv1 srv2 srv3, do-not-place-with pointing at the resource itself. Assembled error reports; count 0 is the only error-report line in the controller log, so again nothing per node.

Ruled out on this run, same as before: the sibling vlstorage-db-vlstorage-generic-0 got all three replicas (UpToDate on srv1, srv2, srv3) in the same window, pools had 127 to 146 GiB free against the 10 requested, all three satellites Online, error report index empty.

Hardware fingerprint from the sandbox-host capture: VM.Standard.E4.Flex, us-ashburn-1, AD-3, FAULT-DOMAIN-1, 128 GB. The 2026-08-18 instance was FAULT-DOMAIN-3 on the same shape and region, so the fault domain is not the shared factor.

Second instance today, run 33687257058 (PR lane, 22:08Z to 22:44Z), same mechanism: vlstorage-db-vlstorage-generic-1 Pending for 19 minutes, 13 CreateVolume cycles refused with Not enough available nodes, the sibling generic-0 fully placed on all three nodes in the same window, no per-node error report. Fingerprint from the sandbox-host capture is identical to the morning instance: VM.Standard.E4.Flex, us-ashburn-1, AD-3, FAULT-DOMAIN-1, 128 GB. Two of today's two instances on the same shape, region and fault domain, against the 2026-08-18 one on FAULT-DOMAIN-3; still too few to call a bias, but the capture is doing its job.

Root cause found. Fixed on main by cee9065 (2026-09-18), which backports LINBIT linstor-server PR #528 (in upstream 1.35.1).

When a storage pool is created, the satellite reads the pool's block size from a temporary zvol. LINSTOR 1.33.3 didn't wait for udev to create the /dev/zvol link, and it swallowed the read error. The node was then left without StorDriver/internal/minIoSize and treated as 512, while the other nodes got 4096. A replicated PVC whose first replica lands on a 512 node gets DrbdOptions/Disk/block-size=512, and placement then rejects every 4096 pool for the second and third replicas (StorPoolFilter). That rejection is logged only at TRACE, so the logs show just "Not enough available nodes", and vlstorage's PVC never gets its replicas.

The LINSTOR database in the captured snapshots shows it:

4 of 4 failing runs (34667295001, 33645509783, 33687257058, 32350576394) have a node with no minIoSize, and that node holds the stuck PVC's first replica in every cycle.
4 of 4 green runs have 4096 on all three pools and on every volume definition.

Ruled out:

a leftover replica (each cycle creates a new resource definition);
free space (105-154 GiB per pool);
node state (all Online);
an image change (red and green ran the same satellite image);
host load.

The PR lane hasn't been able to hit this since 186547d moved it to the local storage class. Only the replicated lane could. Clusters on a release image built before cee9065 can still get unequal pools on install, until a release ships the patch. When linstor-server is bumped to 1.35.1 or later, drop the patch.

Navigation Menu

Repository navigation

bug(tests): e2e install phase flakes under parallel load, always surfacing as a monitoring-system timeout on VLCluster #3851

Status (2026-08-16, supersedes the framing below where they disagree)

What happens

What sits underneath, and what is not yet proven

Load correlation, unproven but consistent

What the recent fixes do not touch

Artifacts available for investigation

What a fix needs

The common object

Mechanism A: LINSTOR autoplace refuses two of three nodes (runs 31827081398 attempt 1, 31939731000)

Mechanism B: volume bound, but no filesystem on it (run 31873151114)

Co-symptoms that are not in the chain

Never, not slowly

Ratio

Two corrections

What the install actually asks of LINSTOR

The refusal is not a capacity report race

What the refusal is not, either

Where this leaves it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bug(tests): e2e install phase flakes under parallel load, always surfacing as a monitoring-system timeout on VLCluster #3851

Description

Status (2026-08-16, supersedes the framing below where they disagree)

What happens

What sits underneath, and what is not yet proven

Load correlation, unproven but consistent

What the recent fixes do not touch

Artifacts available for investigation

What a fix needs

Activity

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

The common object

Mechanism A: LINSTOR autoplace refuses two of three nodes (runs 31827081398 attempt 1, 31939731000)

Mechanism B: volume bound, but no filesystem on it (run 31873151114)

Co-symptoms that are not in the chain

Never, not slowly

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

Ratio

Two corrections

What the install actually asks of LINSTOR

The refusal is not a capacity report race

What the refusal is not, either

Where this leaves it

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 16, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Aug 18, 2026

lexfrei commented on Aug 20, 2026

lexfrei commented on Sep 2, 2026

lexfrei commented on Sep 2, 2026

lexfrei commented on Oct 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions