You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
e2e: the replicated-lane cleanup needs 90 GiB free per node, so volumes left by earlier suites fail the kubernetes suites #4617
In the replicated storage lane, the kubernetes suites fail their cleanup whenever an earlier suite left volumes on the sandbox, even when the kubernetes suite itself reclaimed everything it used. The nightly hits this: in run 36659663201 (2026-09-30) both kubernetes-previous and kubernetes-latest went red on it, and the 2026-09-14 nightly failed the same way.
The cleanup calls cozy_wait_linstor_pool_reclaimed 90 300 (hack/e2e-chainsaw/_lib/run-kubernetes.sh). In the local lane that compares against a per-suite baseline. In the replicated lane it falls through to cozy_wait_linstor_pool_free 90 300, an absolute floor of 90 GiB free on every node. That floor has been blocking since 186547d; before that the call ended in || true.
In the 2026-09-30 run the pool table printed after each kubernetes cleanup is identical, srv1 88.17, srv2 83.09, srv3 127.79 GiB free. So neither suite leaked anything. The space is held by PVCs that are still Bound: the platform ones and about 40 GiB of kafka broker volumes left in tenant-test (#4616). Without the kafka volumes srv2 would be at about 103 GiB and pass.
It also hurts before cleanup. With srv2 at 83 GiB, the kubernetes-latest run could not place the 21 GiB CDI scratch replica for its second worker on srv2 (zfs create -V 22234816KB exit 1 in the ProvisioningFailed events). That worker never came up, the MachineHealthCheck replaced it, and the replacement missed the node-join budget. The comment at run-kubernetes.sh:440-452 describes this exact chain.
Fix shape: record the pre-suite baseline in the replicated lane too and compare against it on cleanup, the local-lane helpers already do this. If an absolute floor is still wanted, check it before the suite starts, where a low pool can be reported as the environment and not as the suite's fault.
The PR lanes set COZY_E2E_STORAGE_CLASS: local, so this only shows up in nightly and in the release-branch lane.
Activity
fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baselin…