fix(reloader): add startupProbe to survive slow first start under load by lexfrei · Pull Request #3470 · cozystack/cozystack

Aleksei Sviridkin (lexfrei) · 2026-07-28T10:02:33Z

What this PR does

Adds a startup probe to the reloader deployment, rendered from package values through a patch on the vendored chart.

Reloader went into CrashLoopBackOff on a loaded install. Events show Liveness probe failed: ... connect: connection refused, repeating on the liveness cadence until the kubelet gave up.

A refusal rather than a timeout closes two questions at once. It is an RST from the target netns, so the packet arrived and the pod network is fine. It also rules out the mode in stakater/Reloader#252, where the listener is already up and /metrics just answers slowly; that one produces probe timeouts. Nothing was listening on :9090 yet.

http.ListenAndServe is the last statement of startReloader, so the whole prologue runs before the socket opens. The slow part is not the informers, those are lazy and do their List in a goroutine. It is IsOpenshift = isOpenshift() in pkg/kube: a package-level var, so it runs at init before main(), and it does a GET /apis/project.openshift.io on a context.TODO() carrying no deadline. The latency is the API server's, and the container waits it out under a 100m CPU limit.

Without a startup probe the only grace period is the liveness budget itself, initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5. A start that overruns it gets killed mid-prologue, and the restart re-runs the prologue from the top against an API server that is no less busy, so the loop feeds itself.

This reaches past reloader. packages/core/platform/sources/linstor.yaml lists cozystack.reloader in the LINSTOR stack's dependsOn, and dependsOn fans out to every component of that stack, so a reloader stuck in this loop blocks that whole install.

The probe is /live at periodSeconds 5 and failureThreshold 45, about 225s, well inside the 10m install timeout that cozystack-operator stamps on generated HelmReleases. Liveness is suspended while it runs and gets its tight budget back as soon as the listener binds. timeoutSeconds is 5 to match the other two probes instead of the kubelet default of 1s, since a throttled process at 100m can hold the socket open and still need more than a second to answer.

Both liveness knobs the chart already exposes would work without any patch. Widening failureThreshold buys the grace permanently and loosens hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, including one that binds in three seconds, and nothing restarts a container that wedges inside it. Adaptive against fixed is the trade. The win is smaller than it sounds while reloader.enableHA is unset, since /live is a constant 200 then and liveness only catches the listener dying outright.

A chart bump does not fix this. 2.2.14 still renders no startupProbe anywhere, and v1.4.19 added a second eager package-level call next to IsOpenshift, so the prologue got longer instead of shorter. packages/system/velero already carries a startupProbe patch on its vendored chart for the same reason.

make update now drives the shared %-update rule with an exact CHART_VERSION. It has to be exact. With a stale local repo cache helm pull --version 2.0.0 fails with version-not-found, while --version "^2.0.0" quietly resolves to 2.2.14, whose deployment.yaml has the ports block at line 179 instead of 161, so the hunk would fuzz or land somewhere unintended.

The patch runs with --forward because patch(1) defaults diverge between implementations over a tree that already carries the change. GNU 2.7.6 refuses and exits 1. Apple 2.0-12u11 takes its Assume -R? [y] default, strips the hunk back out and exits 0. GNU under --batch does the same, which is the trap, since --batch is what you reach for to make a script non-interactive. --forward refuses in all three and leaves a .rej.

Two things keep the probe from disappearing quietly, and they fire at different times. --forward catches it at the moment someone re-runs the patch by hand. The test only catches it at the next run of the suite, late enough for a commit to travel in between.

make test also checks that the vendored Chart.yaml still records the pinned version, so bumping CHART_VERSION without re-running make update fails instead of leaving the pin and the tree disagreeing. Checked by doing the violation and watching it go red.

Screenshots

Not applicable, no UI change.

Downstream repositories

Walked the trigger map in docs/agents/contributing.md against the diff. All five files sit inside one packages/system package. No package is added, renamed or removed, nothing under hack/ is touched, and no CRD, namespace, packages/core values key, version enum or release asset changes.

Release note

fix(reloader): add a startup probe so a slow first start is not killed by the liveness probe and left in CrashLoopBackOff, which also unblocks the LINSTOR stack that depends on reloader

Summary by CodeRabbit

New Features
- Added a configurable startup health probe for the Reloader service.
- The probe allows additional time for the service to start before health checks begin.
- Documented the probe’s timing and behavior in the chart configuration.
Bug Fixes
- Preserved existing liveness and readiness probe behavior.
Tests
- Added coverage for enabled, disabled, and unaffected health probes.
- Added validation to ensure the chart version matches the pinned release.

Reloader binds its :9090 listener as the last statement of startReloader, so /live and /metrics answer connection refused until everything before it has finished. The blocking part of that prologue is the package-level IsOpenshift = isOpenshift() in pkg/kube, which issues a GET /apis/project.openshift.io against the API server on a context carrying no deadline, before main() is even entered. That latency belongs to the API server, and the container waits it out at a 100m CPU limit. The chart renders no startupProbe, so the only grace period is the liveness budget: initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about 60 seconds. A first start that needs longer is killed mid-prologue, and the restart does not resume it, it runs it again from the beginning, so the loop sustains itself. The blast radius reaches past reloader: the LINSTOR stack lists cozystack.reloader in dependsOn, which fans out to every component of that stack, so a reloader stuck in this loop blocks the whole install. Render a values-driven startupProbe on /live so liveness only begins evaluating once the listener is up. Budget = periodSeconds 5 times failureThreshold 45, so about 225 seconds. timeoutSeconds matches the 5s the chart already gives liveness rather than the kubelet default of 1s, which a throttled process can exceed while holding the socket open. Both liveness knobs the chart already exposes could buy the same grace with no patch to carry. Widening failureThreshold buys it permanently, loosening hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, whereas a startup probe returns liveness to its tight budget the moment the listener binds. The chart is vendored, so the template change is carried as patches/deployment-startup-probe.diff and reapplied by `make update`, which now drives the shared %-update rule with an exact chart version pin instead of pulling whatever is latest. The patch is applied with --forward because patch(1) defaults diverge by implementation over a tree that already carries the change: GNU 2.7.6 refuses and exits 1, Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk back out with exit 0, and GNU under --batch does the same. --forward refuses in all three. A helm unittest pins the probe, its budget, the no-op path the template promises for an unset value, and the liveness numbers the budget is sized against, so the documented arithmetic cannot drift away from the chart without a red test. The same target asserts that the vendored Chart.yaml still records the pinned version, so raising the pin without re-running `make update` fails instead of leaving the tree and the pin quietly disagreeing. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>

gemini-code-assist · 2026-07-28T10:02:36Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai · 2026-07-28T10:02:41Z

📝 Walkthrough

Walkthrough

The Reloader Helm chart is pinned to version 2.0.0, gains a configurable startup probe, applies the change through a maintained patch, and adds tests for rendering and probe preservation.

Changes

Reloader startup probe

Layer / File(s)	Summary
Pin chart source and apply patch `packages/system/reloader/Makefile`	The Makefile pins chart version `2.0.0`, validates the vendored chart version, and reapplies the startup probe patch during updates.
Configure and render startup probe `packages/system/reloader/values.yaml`, `packages/system/reloader/patches/deployment-startup-probe.diff`, `packages/system/reloader/charts/reloader/templates/deployment.yaml`	Values define the `/live` HTTP startup probe, and the Deployment template renders it only when configured.
Validate probe behavior `packages/system/reloader/tests/deployment_startup_probe_test.yaml`	Helm tests verify probe fields, omission when unset, and unchanged liveness and readiness probes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: androndo

Sequence Diagram(s)

sequenceDiagram
  participant Values as values.yaml
  participant Template as Deployment template
  participant Deployment as Rendered Deployment
  participant Tests as Helm unit tests
  Values->>Template: provide startupProbe configuration
  Template->>Deployment: render conditional startupProbe
  Tests->>Deployment: assert startup, liveness, and readiness probes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding a startupProbe to the Reloader chart to handle slow first starts.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/reloader-startup-probe

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist · 2026-07-28T11:11:51Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/system/reloader/Makefile`:
- Line 31: Update the Makefile’s update target so it depends only on clean, then
invoke reloader-update from the update recipe after clean completes. Preserve
the existing clean and reloader-update behavior while ensuring parallel make
cannot run them concurrently.

In `@packages/system/reloader/patches/deployment-startup-probe.diff`:
- Around line 16-33: Remove the startupProbe template mutation from
packages/system/reloader/patches/deployment-startup-probe.diff (lines 16-33),
stop applying that patch in packages/system/reloader/Makefile (lines 31-39), and
restore the upstream deployment template verbatim in
packages/system/reloader/charts/reloader/templates/deployment.yaml (lines
164-174). Use a repository-owned overlay or post-rendering mechanism for probe
injection instead.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: aaf6e2df-dc0d-4ccc-aed5-6165e9793034

📥 Commits

Reviewing files that changed from the base of the PR and between 8bffa34 and 7f67d02.

📒 Files selected for processing (5)

packages/system/reloader/Makefile
packages/system/reloader/charts/reloader/templates/deployment.yaml
packages/system/reloader/patches/deployment-startup-probe.diff
packages/system/reloader/tests/deployment_startup_probe_test.yaml
packages/system/reloader/values.yaml

coderabbitai · 2026-07-28T11:15:56Z

+	  exit 1; \
+	fi
+
+update: clean reloader-update


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash set -euo pipefail echo "== Makefile ==" if [ -f packages/system/reloader/Makefile ]; then cat -n packages/system/reloader/Makefile else echo "missing packages/system/reloader/Makefile" fi echo echo "== reloader-update target references ==" rg -n "reloader-update|charts/|patches/|clean:" -S .

Repository: cozystack/cozystack

Length of output: 50376

🏁 Script executed:

#!/bin/bash set -euo pipefail echo "== find relco/Makefile and relevant package.mk ==" fd -a '(^|/)Makefile$|package\.mk$' . | sed 's#^\./##' | rg 'packages/system/reloader/Makefile|hack/package.mk|package.mk' || true echo echo "== package.mk excerpt ==" if [ -f hack/package.mk ]; then sed -n '1,140p' hack/package.mk | cat -n fi echo echo "== reloader Makefile exact section ==" sed -n '31,80p' packages/system/reloader/Makefile | cat -n

Repository: cozystack/cozystack

Length of output: 2537

🏁 Script executed:

#!/bin/bash set -euo pipefail echo "== check GNU make availability and behavior for prerequisite ordering ==" if command -v make >/dev/null 2>&1; then make --version | head -n 1 mkdir -p /tmp/make-dir-race-check && cd /tmp/make-dir-race-check rm -rf charts printf 'clean:\n\trm -rf charts\n\nreloader-update:\n\tmkdir -p charts\n\techo created charts during reloader-update >> status\n\nupdate: clean reloader-update\n\techo updated after prerequisites >> status\n' > Makefile echo "parallel run:" make update -j4 2>&1 || true if [ -e charts ]; then echo "post-update charts exists"; fi if [ -e status ]; then cat status; fi printf 'clean:\n\trm -rf charts\n\nreloader-updatedepended: clean\n\tmkdir -p charts\n\techo created charts after clean >> status\n\nupdate: reloader-updatedepended\n\techo patched using chart >> status\n' > Makefile echo "serialized dependency run:" rm -rf charts status make update -j4 2>&1 || true if [ -e status ]; then cat status; fi else echo "make unavailable" fi

Repository: cozystack/cozystack

Length of output: 247

Serialize clean before reloader-update.

update: clean reloader-update allows make -j to run sibling prerequisite recipes concurrently; clean’s rm -rf charts/ can race with the parallel % -update recipe that populates charts/, so the later patch can see a partial vendored tree. Run reloader-update from the recipe after clean.

Proposed fix

-update: clean reloader-update +update: clean + $(MAKE) reloader-update # Re-apply local modifications carried as patches/*.diff. clean runs first,

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

update: clean reloader-update

update: clean

$(MAKE) reloader-update

🧰 Tools

🪛 checkmake (0.3.2)

[warning] 31-31: Target body for "update" exceeds allowed length of 5 lines (8).

(maxbodylength)

[warning] 31-31: Required target "all" is missing from the Makefile.

(minphony)

[warning] 31-31: Required target "clean" is missing from the Makefile.

(minphony)

[warning] 31-31: Required target "test" must be declared PHONY.

(minphony)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/reloader/Makefile` at line 31, Update the Makefile’s update target so it depends only on clean, then invoke reloader-update from the update recipe after clean completes. Preserve the existing clean and reloader-update behavior while ensuring parallel make cannot run them concurrently.

coderabbitai · 2026-07-28T11:15:56Z

+diff --git a/charts/reloader/templates/deployment.yaml b/charts/reloader/templates/deployment.yaml
+--- a/charts/reloader/templates/deployment.yaml
+++ b/charts/reloader/templates/deployment.yaml
+@@ -161,6 +161,17 @@
+         ports:
+         - name: http
+           containerPort: 9090
+        {{- /*
+        cozystack (patches/deployment-startup-probe.diff): the upstream chart
+        renders no startupProbe, which leaves a slow first start no grace period
+        of its own. Why that matters and how the budget was chosen is documented
+        next to the knob, in the package values.yaml. Rendered with `with`, so an
+        unset value emits nothing and the template stays upstream-compatible.
+        */}}
+        {{- with .Values.reloader.deployment.startupProbe }}
+        startupProbe:
+          {{- toYaml . | nindent 10 }}
+        {{- end }}


📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Keep the vendored chart verbatim.

This patch intentionally reintroduces a local change under packages/**/charts/**, so the vendored tree is no longer an upstream copy. Move probe injection to a repository-owned overlay or post-rendering mechanism instead.

packages/system/reloader/patches/deployment-startup-probe.diff#L16-L33: remove the template-mutating patch.

packages/system/reloader/Makefile#L31-L39: stop applying the local patch during chart updates.

packages/system/reloader/charts/reloader/templates/deployment.yaml#L164-L174: restore the upstream template verbatim.

As per coding guidelines, “Do not edit vendored upstream chart files directly; they must remain verbatim because make update regenerates them.”

📍 Affects 3 files

packages/system/reloader/patches/deployment-startup-probe.diff#L16-L33 (this comment)

packages/system/reloader/Makefile#L31-L39

packages/system/reloader/charts/reloader/templates/deployment.yaml#L164-L174

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/reloader/patches/deployment-startup-probe.diff` around lines 16 - 33, Remove the startupProbe template mutation from packages/system/reloader/patches/deployment-startup-probe.diff (lines 16-33), stop applying that patch in packages/system/reloader/Makefile (lines 31-39), and restore the upstream deployment template verbatim in packages/system/reloader/charts/reloader/templates/deployment.yaml (lines 164-174). Use a repository-owned overlay or post-rendering mechanism for probe injection instead.

Source: Coding guidelines

Andrei Kvapil (kvaps)

LGTM — verified. The root cause is real: reloader binds its :9090 listener as the last statement of startReloader, after an eager package-level isOpenshift() GET issued on a no-deadline context before main(), so a slow first start under load is refused and killed by liveness (~60s), and the restart re-runs the prologue from scratch — a self-sustaining loop. The startupProbe (periodSeconds 5 × failureThreshold 45 ≈ 225s, well inside the 10m install timeout) gives the grace; the template is guarded by with so it stays upstream-compatible, carried as a patch with a version-pin check and a helm unittest. The red E2E is the known node-join flake.

…w first start under load (#4433) Backport of #3470 to `release-1.6`. On the E2E run of #4052 install failed because reloader crash-looped (6 restarts) and `failed early due to stalled resources: [Deployment/cozy-reloader/reloader-reloader status: 'Failed']` blocked the whole linstor chain behind it. Same shape as what #3470 fixes on main, slow first start under load killed by the probes. This one changes the chart, not only CI: reloader gets a startupProbe. Cherry-picked clean with `-x`. ### Testing - `make unit-tests` green (includes the reloader helm unit tests from #3470), POSIX sh sweep clean. ```release-note fix(reloader): add startupProbe to survive slow first start under load ```

github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Jul 28, 2026

Aleksei Sviridkin (lexfrei) added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) and removed area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review labels Jul 28, 2026

Aleksei Sviridkin (lexfrei) marked this pull request as ready for review July 28, 2026 11:11

Aleksei Sviridkin (lexfrei) requested a review from Andrei Kvapil (kvaps) as a code owner July 28, 2026 11:11

Aleksei Sviridkin (lexfrei) requested review from IvanHunters, Andrey Kolkov (androndo), Timofei Larkin (lllamnyp), myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners July 28, 2026 11:11

dosubot Bot added the area/testing Issues or PRs related to testing (e2e, bats, unit tests) label Jul 28, 2026

coderabbitai Bot reviewed Jul 28, 2026

View reviewed changes

Andrei Kvapil (kvaps) approved these changes Jul 28, 2026

View reviewed changes

Andrei Kvapil (kvaps) merged commit d598e67 into main Jul 28, 2026
20 of 21 checks passed

Andrei Kvapil (kvaps) deleted the fix/reloader-startup-probe branch July 28, 2026 15:45

Aleksei Sviridkin (lexfrei) self-assigned this Jul 29, 2026

coderabbitai Bot mentioned this pull request Aug 7, 2026

fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy #3546

Merged

1 task done

myasnikovdaniil mentioned this pull request Sep 24, 2026

[Backport release-1.6] fix(reloader): add startupProbe to survive slow first start under load #4433

Merged

Navigation Menu

Repository navigation

fix(reloader): add startupProbe to survive slow first start under load - #3470

fix(reloader): add startupProbe to survive slow first start under load#3470
Andrei Kvapil (kvaps) merged 1 commit into
mainfrom
fix/reloader-startup-probe

Aleksei Sviridkin (lexfrei) commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 28, 2026

Uh oh!

coderabbitai Bot Jul 28, 2026

Uh oh!

Andrei Kvapil (kvaps) left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	update: clean reloader-update
	update: clean
	$(MAKE) reloader-update

Conversation

Aleksei Sviridkin (lexfrei) commented Jul 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Screenshots

Downstream repositories

Release note

Summary by CodeRabbit

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Andrei Kvapil (kvaps) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Aleksei Sviridkin (lexfrei) commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading