Repository navigation
fix(mariadb): widen startup probe budget so bootstrap cannot be killed - #3344
Conversation
The operator builds the MariaDB startup probe without setting failureThreshold, so it falls back to the Kubernetes default of 3. The kubelet only starts probing once initialDelaySeconds has elapsed and kills the container on the Nth consecutive failure, so with the operator's initialDelaySeconds of 20 and periodSeconds of 10 the budget is 20 + (3 - 1) * 10 = 40 seconds -- exactly what the liveness probe allows, which leaves the startup probe deferring nothing. Bootstrapping a fresh datadir does not reliably fit in 40 seconds under a constrained CPU limit or slow replicated storage. Overrunning the budget is not a slow start that recovers on the next attempt: the kubelet kills the container after mariadb-install-db has created the system tables but before the entrypoint's temporary server applies the root grant. The restarted entrypoint keys DATABASE_ALREADY_EXISTS off that directory and skips initialization for good, so the server comes up with the install-time root account -- local only and passwordless -- rather than the credentials the operator holds. Every probe from then on fails with "Access denied for user 'root'", so the pod crash-loops permanently with no path back and the install times out waiting for a MariaDB that can never become ready. Raise failureThreshold to 30, giving first boot 310 seconds. Only the threshold is set, so the operator keeps its own probe handler -- the agent HTTP endpoint when replication is enabled, the exec check otherwise -- and the liveness and readiness probes keep their defaults, so a server that hangs after a successful start is still caught as quickly as before. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The suite allowed 3 minutes for a MariaDB endpoint address to appear, which was set when a replica had 40 seconds to finish its first boot. The startup probe now grants 310 seconds, so a healthy install can legitimately take longer than the gate allows and would be failed while the probe is still willing to wait. The endpoint assert is the gate that matters here. The chart hands the operator a MariaDB custom resource and ships no built-in workload, so helm has nothing to wait on and the HelmRelease reports Ready as soon as the manifests are applied, long before a server exists. An endpoint address, by contrast, appears only once a replica has passed its startup and then its readiness probe, so that assert is what actually covers bootstrap. Raise it to 10m in both the replicated and the single-replica test. One ready address satisfies the assert, so it has to clear a single startup budget rather than one per replica, plus the operator reconcile, PVC bind and image pull that precede it. The HelmRelease ceilings stay at 6m, since nothing about them gates on the server coming up. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical issue where the MariaDB container could be prematurely killed by the kubelet during its initial bootstrap phase. By extending the startup probe's failure threshold, the operator now allows sufficient time for the database to complete its internal setup, preventing permanent datadir corruption and subsequent crash-looping. The changes ensure robust initialization without compromising the responsiveness of liveness and readiness checks once the server is operational. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
github-actions
Bot
added
area/database
dosubot
Bot
added
area/testing
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughChangesMariaDB startup and readiness
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request increases the startup probe budget for MariaDB to prevent permanent crash-loops during fresh datadir bootstraps on slow storage or constrained CPU environments. Specifically, it configures 'startupProbe.failureThreshold' to 30 (extending the budget to 310 seconds) in the MariaDB template, adds a corresponding Helm unit test suite to verify this behavior, and increases the E2E chainsaw test assert timeouts from 3 minutes to 10 minutes to accommodate the longer startup budget. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
myasnikovdaniil
left a comment
myasnikovdaniil
left a comment
There was a problem hiding this comment.
LGTM — the root cause is correctly diagnosed and the fix is minimal: it sets only failureThreshold, which the operator merges into its own probe rather than replacing it.
Business context: The operator's default MariaDB startup probe gets a 40s budget — identical to the liveness budget it exists to defer — so on a slow first boot the kubelet kills the container mid-mariadb-install-db, the restarted entrypoint sees a populated datadir and skips the root grant, and the pod crash-loops permanently; this widens the startup budget to 310s.
Verified against the operator source at the pinned version (25.10.2):
- Default budget is 40s as described.
defaultProbesetsinitialDelaySeconds: 20, periodSeconds: 10with nofailureThreshold(→ Kubernetes default 3), i.e.20 + (3-1)*10 = 40s. Before this change the template set nostartupProbe, so it deferred nothing. - Setting only
failureThresholdpreserves the operator's handler.mariadbProbereturns the user probe verbatim only when itsProbeHandleris non-empty. With no handler set here, the operator deep-copies its own default (standalone:exec … SELECT 1; replicated: agenthttpGet /liveness) and then appliesfailureThreshold: 30viasetProbeThresholds, keepinginitialDelaySeconds: 20andperiodSeconds: 10→20 + 29*10 = 310s. This is exactly what the newnotExistsassertions onspec.startupProbe.{exec,httpGet,tcpSocket}protect. - Liveness/readiness stay on operator defaults, so a server that hangs after a successful start is still caught on the ~40s liveness budget — no regression to fail-fast.
- e2e timeout change is consistent. The chart ships only the MariaDB CR (no built-in workload), so Helm's wait cannot block on it and the HelmRelease goes Ready on apply; the Endpoints assert is the first gate that waits on a booted replica, and at 3m it sat below the new 310s budget. Raising it to 10m (both cases) is right, and leaving the 6m HelmRelease ceiling unchanged is right for the same reason.
Nit (non-blocking): the template comment says mysql_install_db while the commit body says mariadb-install-db — same tool, just worth keeping consistent.
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-3344-to-release-1.5
git worktree add --checkout .worktree/backport-3344-to-release-1.5 backport-3344-to-release-1.5
cd .worktree/backport-3344-to-release-1.5
git reset --hard HEAD^
git cherry-pick -x 1e4a1522913aaada1d9df7ed364d87f08b2ef713
git push --force-with-lease |
Backport of #3344 onto release-1.5. The bot's cherry-pick imported hack/e2e-chainsaw/mariadb/chainsaw-test.yaml whole, because release-1.5 has no such file: the app suite here is still BATS, and the Chainsaw port landed with #2826 on main only. That file is dropped and its widening applied to the gate that actually runs on this branch instead. That gate needed the change more than the chainsaw one did. In hack/e2e-apps/mariadb.bats both tests wait 80s for an endpoint address to appear, and an address appears only once a replica has cleared its startup then readiness probe. This commit raises the startup budget to 310s, so an 80s ceiling would fail runs the probe was still willing to wait for -- importing the chainsaw file and leaving the BATS suite alone would have turned the fix into an e2e failure. Both waits go to 600s, matching the 10m the upstream chainsaw assert uses and for the same reason. packages/apps/mariadb/Makefile gains the `test:` target. It is absent on release-1.5 -- it arrived with #3071, which was not backported -- and hack/helm-unit-tests.sh discovers suites by probing for that target, so without it the backported tests/startup_probe_test.yaml would never be executed by `make unit-tests`. Verified locally: 4 assertions in tests/startup_probe_test.yaml pass. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit a37715062e5da24e54956f075dd2d6417c4dff2e)
2 participants
What this PR does
The operator builds the MariaDB startup probe without setting
failureThreshold, so it falls back to the Kubernetes default of 3. The kubelet only starts probing afterinitialDelaySecondsand kills the container on the Nth consecutive failure, so the budget isinitialDelay + (failureThreshold - 1) * period— with the operator's 20s delay and 10s period that is 40 seconds, exactly the budget of the liveness probe the startup probe exists to defer.Bootstrapping a fresh datadir does not reliably finish inside 40 seconds under a constrained CPU limit or slow replicated storage, and overrunning it is not a slow start that recovers on the next attempt. The kubelet kills the container after
mariadb-install-dbhas created the system tables but before the entrypoint's temporary server applies the root grant. The restarted entrypoint keysDATABASE_ALREADY_EXISTSoff that directory, skips initialization for good, and the server comes up with the install-time root account — local only and passwordless — instead of the credentials the operator holds. Every probe from then on fails withAccess denied for user 'root', the pod crash-loops permanently, and the install times out on a MariaDB that can never become ready.Raising
failureThresholdto 30 lifts the first-boot budget to 310 seconds. Only the threshold is set, so the operator keeps its own probe handler — the agent HTTP endpoint when replication is enabled, the exec check otherwise. Liveness and readiness keep the operator defaults and stay suppressed until the startup probe first succeeds, so a server that hangs after a successful start is still caught as quickly as before.The defect is still present in the latest upstream operator release, so a version bump does not resolve it.
The e2e suite needed a matching adjustment. The chart hands the operator a
MariaDBCR and ships no workload of its own, so the HelmRelease goes Ready as soon as helm applies the manifests and never waits for a pod — the gate that actually covers first boot is the Endpoints assert, which sat at 3m, below the budget the probe now grants. That one moves to 10m; the HelmRelease ceilings are left as they were.Release note
Summary by CodeRabbit
Bug Fixes
Tests