fix(kamaji): increase memory limits and add startup probe by IvanHunters · Pull Request #2421 · cozystack/cozystack

IvanHunters · 2026-04-19T16:02:57Z

Summary

Increase kamaji controller memory limit from 500Mi to 512Mi
Increase kamaji controller memory request from 100Mi to 256Mi
Add startup probe with 60-second timeout (12 attempts × 5s periods)
Increase readiness/liveness probe initialDelaySeconds from 5s/15s to 30s

Problem

The kamaji controller was experiencing frequent CrashLoopBackOff due to OOMKilled errors. Analysis showed:

Container was being killed with exit code 137 (OOMKilled) after ~20-25 seconds of runtime
Memory limit of 500Mi was insufficient for controller initialization
Readiness probe was failing because it started too early (5s initialDelay), before the controller finished leader election (~17s)

Solution

Memory increase:

Limit: 500Mi → 512Mi (based on production testing)
Request: 100Mi → 256Mi (ensures adequate reservation)

Startup probe:

Added to give controller up to 60 seconds to initialize without being killed by liveness probe
12 attempts × 5s period = 60s maximum startup time

Probe delays:

ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to acquire leader lease)
LivenessProbe: 15s → 30s initialDelay (aligned with readiness)

Testing

Verified in production cluster:

Controller runs stable with 0 restarts
No more OOMKilled events
Successfully creates kubeconfig secrets for tenant clusters

Related Issues

Fixes tenant cluster components stuck in ContainerCreating due to missing kubeconfig secrets (caused by crashing kamaji controller).

Summary by CodeRabbit

Chores
- Introduced automated health checks using HTTP-based probes to monitor service status during startup, continuous operation, and readiness to handle traffic.
- Adjusted container memory resource allocation for enhanced stability and performance.

- Increase memory limit from 500Mi to 512Mi - Increase memory request from 100Mi to 256Mi - Add startup probe with 60s timeout (12 attempts × 5s) - Increase readiness/liveness initialDelaySeconds from 5/15 to 30s This fixes OOMKilled crashes observed in production where kamaji controller was being killed due to insufficient memory during startup. Signed-off-by: IvanHunters <[email protected]>

claude

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.

Once credits are available, reopen this pull request to trigger a review.

gemini-code-assist · 2026-04-19T16:03:04Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses stability issues in the Kamaji controller by adjusting resource constraints and health check configurations. By providing more memory and allowing for a longer initialization window, the changes resolve frequent crash loops caused by insufficient memory and premature health checks during the startup phase.

Highlights

Resource Allocation: Increased the Kamaji controller memory limit to 512Mi and the memory request to 256Mi to prevent OOMKilled errors.
Startup and Health Probes: Introduced a startup probe with a 60-second grace period and adjusted the initial delay for liveness and readiness probes to accommodate controller initialization and leader election.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	Gemini (@gemini-code-assist)	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

coderabbitai · 2026-04-19T16:03:10Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5b7e7ffa-7ef1-4969-95a8-e8203120e956

📥 Commits

Reviewing files that changed from the base of the PR and between e26894e and e148343.

📒 Files selected for processing (1)

packages/system/kamaji/values.yaml

📝 Walkthrough

Walkthrough

Updated Kamaji container memory sizing by increasing request from 100Mi to 256Mi and limit from 500Mi to 512Mi. Introduced HTTP-based health check probes—startup, liveness, and readiness—targeting /healthz and /readyz endpoints on the healthcheck port with specified timing parameters.

Changes

Cohort / File(s)	Summary
Kamaji Health Probes & Resource Configuration `packages/system/kamaji/values.yaml`	Increased container memory requests and limits; added `startupProbe`, `livenessProbe`, and `readinessProbe` with HTTP health check endpoints (`/healthz`, `/readyz`), timeouts, and check intervals.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 A hop, a check, a heartbeat's glow,
Health probes bloom where containers grow,
With readyz and healthz paths so clear,
Kamaji dances without a fear! 🏃‍♂️✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: increasing memory limits and adding a startup probe to the Kamaji controller configuration.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/kamaji-memory-and-probes

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

This pull request updates the resource requests and limits for the Kamaji package and introduces startup, liveness, and readiness probes. The review feedback suggests optimizing the probe configuration by reducing the initialDelaySeconds for liveness and readiness probes, as the new startup probe makes these delays redundant. Additionally, it is recommended to further increase the memory limit to provide better stability against potential OOM events.

gemini-code-assist · 2026-04-19T16:04:02Z

+    httpGet:
+      path: /healthz
+      port: healthcheck
+    initialDelaySeconds: 30


When a startupProbe is configured, initialDelaySeconds on the livenessProbe is typically redundant because the liveness probe only starts after the startup probe has succeeded. Keeping a high initialDelaySeconds here unnecessarily delays the container from being monitored for liveness after it has successfully started. It is recommended to set this to 0 or a very small value.

initialDelaySeconds: 0

gemini-code-assist · 2026-04-19T16:04:02Z

+    httpGet:
+      path: /readyz
+      port: healthcheck
+    initialDelaySeconds: 30


Similar to the liveness probe, initialDelaySeconds on the readinessProbe is redundant when using a startupProbe. The readiness probe will naturally fail until the application is ready (e.g., finished leader election). Hardcoding a 30-second delay here unnecessarily prevents the pod from becoming ready as soon as it's actually capable of serving requests.

initialDelaySeconds: 0

gemini-code-assist · 2026-04-19T16:04:02Z

    limits:
      cpu: 200m
-      memory: 500Mi
+      memory: 512Mi


The memory limit increase from 500Mi to 512Mi is quite small (approx. 2.4%). While the description mentions this is based on production testing, such a tight margin might lead to future OOMKilled events if the controller's memory usage fluctuates slightly. Consider providing a bit more headroom (e.g., 640Mi or 768Mi) for better stability.

github-actions · 2026-04-24T20:12:13Z

Successfully created backport PR for release-1.3:

[Backport release-1.3] fix(kamaji): increase memory limits and add startup probe #2491

…artup probe (#2491) # Description Backport of #2421 to `release-1.3`.

@kvaps

The auto-generated changelog only listed #2459/#2467 (velero-configmap Role move). Five additional PRs were backported and merged into release-1.3 between then and the v1.3.1 tag (2026-04-28): - #2471/#2524 - fix(api): IDOR in TenantNamespace Get/Watch - #2496/#2505 - feat(linstor): linstor-csi v1.10.6 (Protocol-C dual-attach) - #2462/#2511 - fix(etcd): remove destructive post-upgrade hook - #2421/#2491 - fix(kamaji): memory limits + startup probe - #2498/#2518 - build(linstor): wire linstor-gui into root build target Update the release date to match the actual tag (2026-04-28), rewrite the intro paragraph, and add @kvaps to contributors. Signed-off-by: Myasnikov Daniil <[email protected]>

IvanHunters requested review from Andrey Kolkov (androndo), Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp) and Kirill Ilin (sircthulhu) as code owners April 19, 2026 16:02

claude Bot reviewed Apr 19, 2026

View reviewed changes

dosubot Bot added the size/XS This PR changes 0-9 lines, ignoring generated files label Apr 19, 2026

gemini-code-assist Bot reviewed Apr 19, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) approved these changes Apr 19, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 19, 2026

Andrei Kvapil (kvaps) added the kind/backport Categorizes issue or PR as requiring a backport to the current release line label Apr 21, 2026

IvanHunters merged commit 52edffb into main Apr 24, 2026
19 of 21 checks passed

IvanHunters deleted the fix/kamaji-memory-and-probes branch April 24, 2026 20:11

github-actions Bot mentioned this pull request Apr 24, 2026

[Backport release-1.3] fix(kamaji): increase memory limits and add startup probe #2491

Merged

myasnikovdaniil added a commit that referenced this pull request Apr 28, 2026

[Backport release-1.3] fix(kamaji): increase memory limits and add st…

3c2aeb1

…artup probe (#2491) # Description Backport of #2421 to `release-1.3`.

Navigation Menu

Repository navigation

fix(kamaji): increase memory limits and add startup probe - #2421

fix(kamaji): increase memory limits and add startup probe#2421
IvanHunters merged 1 commit into
mainfrom
fix/kamaji-memory-and-probes

IvanHunters commented Apr 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gemini-code-assist Bot commented Apr 19, 2026

Uh oh!

coderabbitai Bot commented Apr 19, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Uh oh!

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

IvanHunters commented Apr 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Testing

Related Issues

Summary by CodeRabbit

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot commented Apr 19, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

coderabbitai Bot commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Apr 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanHunters commented Apr 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 19, 2026 •

edited

Loading