Skip to content

[Backport release-1.6] fix(reloader): add startupProbe to survive slow first start under load - #4433

Merged
myasnikovdaniil merged 1 commit into
release-1.6from
backport-3470-to-release-1.6
Sep 24, 2026
Merged

myasnikovdaniil merged 1 commit into
release-1.6from
backport-3470-to-release-1.6

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Backport of #3470 to release-1.6.

On the E2E run of #4052 install failed because reloader crash-looped (6 restarts) and failed early due to stalled resources: [Deployment/cozy-reloader/reloader-reloader status: 'Failed'] blocked the whole linstor chain behind it. Same shape as what #3470 fixes on main, slow first start under load killed by the probes. This one changes the chart, not only CI: reloader gets a startupProbe.

Cherry-picked clean with -x.

Testing

fix(reloader): add startupProbe to survive slow first start under load

Reloader binds its :9090 listener as the last statement of
startReloader, so /live and /metrics answer connection refused until
everything before it has finished. The blocking part of that prologue
is the package-level IsOpenshift = isOpenshift() in pkg/kube, which
issues a GET /apis/project.openshift.io against the API server on a
context carrying no deadline, before main() is even entered. That
latency belongs to the API server, and the container waits it out at a
100m CPU limit.

The chart renders no startupProbe, so the only grace period is the
liveness budget: initialDelaySeconds 10 + periodSeconds 10 *
failureThreshold 5, about 60 seconds. A first start that needs longer
is killed mid-prologue, and the restart does not resume it, it runs it
again from the beginning, so the loop sustains itself. The blast
radius reaches past reloader: the LINSTOR stack lists
cozystack.reloader in dependsOn, which fans out to every component of
that stack, so a reloader stuck in this loop blocks the whole install.

Render a values-driven startupProbe on /live so liveness only begins
evaluating once the listener is up. Budget = periodSeconds 5 times
failureThreshold 45, so about 225 seconds. timeoutSeconds matches the
5s the chart already gives liveness rather than the kubelet default of
1s, which a throttled process can exceed while holding the socket
open.

Both liveness knobs the chart already exposes could buy the same grace
with no patch to carry. Widening failureThreshold buys it permanently,
loosening hang detection for the life of the container.
initialDelaySeconds is start-up-only and would genuinely do, but it is
a fixed blind window on every start, whereas a startup probe returns
liveness to its tight budget the moment the listener binds.

The chart is vendored, so the template change is carried as
patches/deployment-startup-probe.diff and reapplied by `make update`,
which now drives the shared %-update rule with an exact chart version
pin instead of pulling whatever is latest. The patch is applied with
--forward because patch(1) defaults diverge by implementation over a
tree that already carries the change: GNU 2.7.6 refuses and exits 1,
Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk
back out with exit 0, and GNU under --batch does the same. --forward
refuses in all three.

A helm unittest pins the probe, its budget, the no-op path the
template promises for an unset value, and the liveness numbers the
budget is sized against, so the documented arithmetic cannot drift
away from the chart without a red test. The same target asserts that
the vendored Chart.yaml still records the pinned version, so raising
the pin without re-running `make update` fails instead of leaving the
tree and the pin quietly disagreeing.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit 7f67d02)
Signed-off-by: Myasnikov Daniil <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 992cf23c-cf6a-4f93-be09-f7ac16344660

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Sep 24, 2026
@myasnikovdaniil
myasnikovdaniil merged commit c86dbf5 into release-1.6 Sep 24, 2026
14 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3470-to-release-1.6 branch September 24, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants