Skip to content

[Backport release-1.6] fix(e2e): budget each linstor wait separately in post-install-prep - #4434

Merged
myasnikovdaniil merged 1 commit into
release-1.6from
backport-3715-to-release-1.6
Sep 24, 2026
Merged

myasnikovdaniil merged 1 commit into
release-1.6from
backport-3715-to-release-1.6

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Backport of #3715 to release-1.6.

Both linstor waits in hack/e2e-post-install-prep.sh still run off one 15m deadline on this branch, so a slow linstor HelmRelease eats the Deployment's budget and install reports the wrong link as broken. Each wait now gets its own budget.

Cherry-picked clean with -x.

Testing

  • make unit-tests green, POSIX sh sweep clean.
NONE

The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.

Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.

Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.

It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.

A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.

The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.

Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.

The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.

The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit 86674c7)
Signed-off-by: Myasnikov Daniil <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 329f95f8-5a81-43af-b10b-1a2ab83938ff

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Sep 24, 2026
@myasnikovdaniil
myasnikovdaniil merged commit e037707 into release-1.6 Sep 24, 2026
13 of 14 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3715-to-release-1.6 branch September 24, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants