Skip to content

e2e: the replicated-lane cleanup needs 90 GiB free per node, so volumes left by earlier suites fail the kubernetes suites #4617

Closed

Description

In the replicated storage lane, the kubernetes suites fail their cleanup whenever an earlier suite left volumes on the sandbox, even when the kubernetes suite itself reclaimed everything it used. The nightly hits this: in run 36659663201 (2026-09-30) both kubernetes-previous and kubernetes-latest went red on it, and the 2026-09-14 nightly failed the same way.

The cleanup calls cozy_wait_linstor_pool_reclaimed 90 300 (hack/e2e-chainsaw/_lib/run-kubernetes.sh). In the local lane that compares against a per-suite baseline. In the replicated lane it falls through to cozy_wait_linstor_pool_free 90 300, an absolute floor of 90 GiB free on every node. That floor has been blocking since 186547d; before that the call ended in || true.

In the 2026-09-30 run the pool table printed after each kubernetes cleanup is identical, srv1 88.17, srv2 83.09, srv3 127.79 GiB free. So neither suite leaked anything. The space is held by PVCs that are still Bound: the platform ones and about 40 GiB of kafka broker volumes left in tenant-test (#4616). Without the kafka volumes srv2 would be at about 103 GiB and pass.

It also hurts before cleanup. With srv2 at 83 GiB, the kubernetes-latest run could not place the 21 GiB CDI scratch replica for its second worker on srv2 (zfs create -V 22234816KB exit 1 in the ProvisioningFailed events). That worker never came up, the MachineHealthCheck replaced it, and the replacement missed the node-join budget. The comment at run-kubernetes.sh:440-452 describes this exact chain.

Fix shape: record the pre-suite baseline in the replicated lane too and compare against it on cleanup, the local-lane helpers already do this. If an absolute floor is still wanted, check it before the suite starts, where a low pool can be reported as the environment and not as the suite's fault.

The PR lanes set COZY_E2E_STORAGE_CLASS: local, so this only shows up in nightly and in the release-branch lane.

Activity

  1. added a commit that references this issue on Oct 3, 2026
    89c7ea0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugtriage/needs-triageIndicates an issue needs triage by a maintainer

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions