fix(kubernetes): make MachineHealthCheck maxUnhealthy configurable by mattia-eleuteri · Pull Request #2752 · cozystack/cozystack

mattia-eleuteri · 2026-05-27T10:29:18Z

What this PR does

The per-node-group MachineHealthCheck was rendered with a hardcoded maxUnhealthy: 0. In Cluster API semantics this blocks all remediation (the threshold is exceeded as soon as a single machine is unhealthy), so crashed worker nodes in tenant clusters were never auto-replaced.
Exposes maxUnhealthy as a per-node-group field (integer or percentage), defaulting to "100%", so auto-remediation works out of the box and stays tunable for sensitive pools.
Regenerated values.schema.json, README and the -rd CRD accordingly.

Release note

fix(kubernetes): MachineHealthCheck `maxUnhealthy` is now configurable per node group (default `100%`), restoring worker-node auto-remediation that was previously disabled by a hardcoded value of `0`.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Added a per-node-group maxUnhealthy setting with default "100%".
- Machine health checks now use each node group's maxUnhealthy value instead of a fixed value.
Documentation
- Updated app documentation, examples and chart value schema to include the new maxUnhealthy node group option.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

gemini-code-assist · 2026-05-27T10:29:29Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where auto-remediation for worker nodes in tenant clusters was effectively disabled due to a hardcoded maxUnhealthy value of 0. By making this parameter configurable per node group with a sensible default of 100%, the change restores the expected auto-remediation behavior while providing flexibility for different cluster pool requirements.

Highlights

Configurable MachineHealthCheck: Exposed maxUnhealthy as a configurable field per node group, allowing users to define integer or percentage thresholds for auto-remediation.
Remediation Fix: Replaced the hardcoded maxUnhealthy: 0 (which disabled auto-remediation) with a default value of 100% to ensure worker nodes are correctly auto-replaced when unhealthy.
Documentation and Schema Updates: Updated values.schema.json, README.md, and the relevant CRD to reflect the new maxUnhealthy parameter.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	Gemini (@gemini-code-assist)	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

coderabbitai · 2026-05-27T10:29:32Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c4dd914d-1ab6-4c4e-8b8b-d4be6d9dda1d

📥 Commits

Reviewing files that changed from the base of the PR and between 960a1fb and 6a9047d.

📒 Files selected for processing (5)

api/apps/v1alpha1/kubernetes/types.go
packages/apps/kubernetes/README.md
packages/apps/kubernetes/templates/cluster.yaml
packages/apps/kubernetes/values.schema.json
packages/apps/kubernetes/values.yaml

✅ Files skipped from review due to trivial changes (1)

packages/apps/kubernetes/README.md

🚧 Files skipped from review as they are similar to previous changes (3)

packages/apps/kubernetes/values.yaml
packages/apps/kubernetes/values.schema.json
api/apps/v1alpha1/kubernetes/types.go

📝 Walkthrough

Walkthrough

This PR adds a per-node-group maxUnhealthy setting (string, integer or percentage) with a default of "100%", documents it, templates it into the cluster MachineHealthCheck (quoted), and updates JSON schema, ApplicationDefinition keysOrder, and the API type.

Changes

Configurable maxUnhealthy Field for Worker Node Groups

Layer / File(s)	Summary
Values docs/example and cluster template `packages/apps/kubernetes/values.yaml`, `packages/apps/kubernetes/templates/cluster.yaml`, `packages/apps/kubernetes/README.md`	Documented `maxUnhealthy` for NodeGroup, added `nodeGroups.md0.maxUnhealthy: "100%"` example, and updated MachineHealthCheck template to render `$group.maxUnhealthy` (quoted) with default `"100%"` instead of `0`.
Schema, manifest, and API types `packages/apps/kubernetes/values.schema.json`, `packages/system/kubernetes-rd/cozyrds/kubernetes.yaml`, `api/apps/v1alpha1/kubernetes/types.go`	Added `maxUnhealthy` property (type: string, default `"100%"`) to the node-group schema and defaults, updated embedded OpenAPI `spec.nodeGroups.md0.maxUnhealthy` and `keysOrder`, and added `NodeGroup.MaxUnhealthy string` to the API with kubebuilder default `"100%"`.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers:

lexfrei
androndo
sircthulhu
lllamnyp
IvanHunters
myasnikovdaniil

"I nibble at defaults, soft and bright,
A hundred percent now set just right,
Each node group hums with kinder rules,
No hard zeroes for my cluster tools,
Hop, tweak, apply — the nodes sleep tight." 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and accurately summarizes the main change: making MachineHealthCheck's maxUnhealthy parameter configurable per node group, which is the core purpose of this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Trivy (0.69.3)

Trivy execution timed out

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

This pull request introduces the maxUnhealthy parameter to the worker node groups configuration in the Kubernetes application package. This parameter allows users to specify the maximum number of unhealthy nodes (as an integer or percentage) tolerated before the MachineHealthCheck stops auto-remediation. The changes include updates to the README documentation, the Helm templates, the values schema, default values, and the corresponding CozyRDS definition. One low-severity issue was identified in the Helm template where quoting the rendered maxUnhealthy value is recommended to prevent YAML parsing issues when percentage strings are evaluated.

gemini-code-assist · 2026-05-27T10:30:29Z

 spec:
  clusterName: {{ $.Release.Name }}
-  maxUnhealthy: 0
+  maxUnhealthy: {{ $group.maxUnhealthy | default "100%" }}


Quoting the maxUnhealthy value is highly recommended. In Kubernetes, maxUnhealthy can be either an integer or a percentage string (e.g., "10%"). Unquoted percentage values like 100% or 10% can sometimes cause parsing issues in YAML or be misinterpreted by some parsers. Quoting the rendered value ensures it is always treated as a valid string/scalar.

maxUnhealthy: {{ $group.maxUnhealthy | default "100%" | quote }}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

packages/apps/kubernetes/values.yaml (1)
31-43: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

maxUnhealthy type currently blocks integer values

Line 31 defines maxUnhealthy as string, so generated validation/docs enforce string-only and reject valid integer inputs (for example maxUnhealthy: 1) despite the PR contract saying integer or percentage.

Use an int-or-string type in the source definition so generated schema/CRD accept both forms consistently.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes/values.yaml` around lines 31 - 43, Update the
documented/type annotation for the NodeGroup field `maxUnhealthy` from a plain
string to an int-or-string type so generated schema/CRD accepts both integer and
percentage forms; modify the comment line that currently reads "`@field` {string}
maxUnhealthy" to use an int-or-string type (e.g. "`@field` {int-or-string}
maxUnhealthy" or the project's equivalent int|string union) so the
`nodeGroups`/`NodeGroup` validation allows values like `1` and `"100%"`.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@packages/apps/kubernetes/values.yaml`:
- Around line 31-43: Update the documented/type annotation for the NodeGroup
field `maxUnhealthy` from a plain string to an int-or-string type so generated
schema/CRD accepts both integer and percentage forms; modify the comment line
that currently reads "`@field` {string} maxUnhealthy" to use an int-or-string type
(e.g. "`@field` {int-or-string} maxUnhealthy" or the project's equivalent
int|string union) so the `nodeGroups`/`NodeGroup` validation allows values like
`1` and `"100%"`.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4f27b1ec-6912-4aa8-afb7-0bac975fadad

📥 Commits

Reviewing files that changed from the base of the PR and between b630706 and 702a622.

📒 Files selected for processing (5)

packages/apps/kubernetes/README.md
packages/apps/kubernetes/templates/cluster.yaml
packages/apps/kubernetes/values.schema.json
packages/apps/kubernetes/values.yaml
packages/system/kubernetes-rd/cozyrds/kubernetes.yaml

mattia-eleuteri · 2026-05-27T14:35:50Z

Rebased onto latest main (resolved conflicts in the generated schema/README/CRD by regenerating with make generate).

Addressed review feedback:

Gemini (@gemini-code-assist): maxUnhealthy is now rendered with | quote so both integer and percentage values are emitted as valid scalars.
Also made maxUnhealthy optional in the schema (was incorrectly required): the new helm-unittest suite surfaced that a required field would reject existing node groups that don't set it — which would break existing tenant CRs on upgrade. It now defaults to 100% when omitted. All 124 unit tests pass.

Andrei Kvapil (kvaps)

LGTM, thank you!

myasnikovdaniil · 2026-06-10T12:37:27Z

Heads-up — this has drifted ~200 commits behind main and now conflicts only on the generated file packages/system/kubernetes-rd/cozyrds/kubernetes.yaml. The source (values.yaml, templates/, types.go) merges cleanly.

To resolve:

git rebase origin/main
# take main's side for the generated file, then regenerate it from the merged source:
cd packages/apps/kubernetes && make generate
git add -A && git rebase --continue

make generate re-derives values.schema.json, README.md and the kubernetes-rd cozyrds from your merged values.yaml + types.go, so the conflict disappears. That's the only thing standing between this PR (already approved) and merge.

The MachineHealthCheck was hardcoded to maxUnhealthy: 0. In Cluster API semantics this blocks all remediation: as soon as a single machine is unhealthy the threshold is exceeded, so crashed worker nodes are never auto-replaced. Expose maxUnhealthy per node group (integer or percentage) with a default of "100%" so auto-remediation works out of the box while staying tunable for sensitive pools. Signed-off-by: mattia-eleuteri <[email protected]>

mattia-eleuteri · 2026-06-10T12:48:04Z

myasnikovdaniil rebased onto latest main as suggested — took main's side for the generated cozyrds/kubernetes.yaml and regenerated with cozyvalues-gen v1.6.0 (the new pin). Source merged cleanly; 129 chart unit tests pass.

Aleksei Sviridkin (lexfrei) · 2026-07-03T13:25:15Z

Closing this: configurable worker MHC maxUnhealthy already landed on main. The hard-coded 0 was replaced by nodeHealthCheck.maxUnhealthy (default "50%", IntOrString rendering with a dedicated rendering test), so remediation is configurable out of the box now. This branch also conflicts with main at this point. If per-node-group granularity on top of the merged cluster-level field is still wanted, a fresh PR rebased on the current code would be the way to go. Thanks for pushing on this.

Adds per-node-group overrides for the worker MachineHealthCheck tuning introduced as a cluster-wide `nodeHealthCheck` by #2931. **What:** optional `nodeGroups[name].maxUnhealthy` and `nodeGroups[name].nodeStartupTimeout`; when unset a group inherits the cluster-wide `nodeHealthCheck.*` (no behaviour change). The per-group value reuses the existing int-or-string validation; an invalid override fails the render naming the offending group. **Why:** the MachineHealthCheck is rendered per node group, so groups can warrant different remediation tolerances (e.g. a stateful group at `0%`, stateless at `50%`). This is the per-group granularity of the standalone proposal (#2752 / #2935), rebuilt on top of #2931's `nodeHealthCheck`. **Notes:** stacked on `phase1-talos-migration` (#2931) — retarget to `main` once it merges. Supersedes #2935 (original author credited as co-author). helm-unittest covers per-group override (percent + bare int), `nodeStartupTimeout` override, fallback-when-unset, and invalid-override rejection. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added per-node-group overrides for health-check settings: `maxUnhealthy` and `nodeStartupTimeout`, with automatic fallback to cluster-wide `nodeHealthCheck` when unset. * **Bug Fixes** * Improved `maxUnhealthy` parsing/rendering to support both bare integers and percentage strings, including more precise validation errors for the exact override path. * **Documentation** * Updated kubernetes app docs and values schemas (including CRD chart-values schema) to document the new override fields. * **Tests** * Added MachineHealthCheck rendering tests for override, fallback, and invalid `maxUnhealthy` (e.g., `"abc"`) cases.

mattia-eleuteri requested review from IvanHunters, Andrey Kolkov (androndo), Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners May 27, 2026 10:29

claude Bot reviewed May 27, 2026

View reviewed changes

github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/kubernetes Issues or PRs related to the tenant Kubernetes app labels May 27, 2026

github-actions Bot added the kind/bug Categorizes issue or PR as related to a bug label May 27, 2026

gemini-code-assist Bot reviewed May 27, 2026

View reviewed changes

coderabbitai Bot reviewed May 27, 2026

View reviewed changes

mattia-eleuteri force-pushed the fix/kubernetes-mhc-maxunhealthy branch from 702a622 to 6a3bfc9 Compare May 27, 2026 14:33

github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files and removed size/L This PR changes 100-499 lines, ignoring generated files labels May 27, 2026

mattia-eleuteri force-pushed the fix/kubernetes-mhc-maxunhealthy branch from 6a3bfc9 to 3340c17 Compare May 27, 2026 14:35

github-actions Bot added size/S This PR changes 10-29 lines, ignoring generated files and removed size/M This PR changes 30-99 lines, ignoring generated files labels May 27, 2026

mattia-eleuteri force-pushed the fix/kubernetes-mhc-maxunhealthy branch from 3340c17 to 960a1fb Compare May 27, 2026 14:58

Andrei Kvapil (kvaps) previously approved these changes May 28, 2026

View reviewed changes

mattia-eleuteri dismissed Andrei Kvapil (kvaps)’s stale review via 6a9047d June 10, 2026 12:43

mattia-eleuteri force-pushed the fix/kubernetes-mhc-maxunhealthy branch from 960a1fb to 6a9047d Compare June 10, 2026 12:43

myasnikovdaniil mentioned this pull request Jun 16, 2026

fix(kubernetes): make MachineHealthCheck maxUnhealthy configurable #2935

Closed

myasnikovdaniil mentioned this pull request Jun 25, 2026

feat(kubernetes): per-node-group nodeHealthCheck overrides #3053

Merged

Aleksei Sviridkin (lexfrei) closed this Jul 3, 2026

Navigation Menu

Repository navigation

fix(kubernetes): make MachineHealthCheck maxUnhealthy configurable - #2752

fix(kubernetes): make MachineHealthCheck maxUnhealthy configurable#2752
mattia-eleuteri wants to merge 1 commit into
cozystack:mainfrom
mattia-eleuteri:fix/kubernetes-mhc-maxunhealthy

mattia-eleuteri commented May 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

claude Bot left a comment

Uh oh!

gemini-code-assist Bot commented May 27, 2026

Uh oh!

coderabbitai Bot commented May 27, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 27, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

mattia-eleuteri commented May 27, 2026

Uh oh!

Andrei Kvapil (kvaps) left a comment

Uh oh!

myasnikovdaniil commented Jun 10, 2026

Uh oh!

mattia-eleuteri commented Jun 10, 2026

Uh oh!

Aleksei Sviridkin (lexfrei) commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mattia-eleuteri commented May 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Release note

Summary by CodeRabbit

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gemini-code-assist Bot commented May 27, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

coderabbitai Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mattia-eleuteri commented May 27, 2026

Uh oh!

Andrei Kvapil (kvaps) left a comment

Choose a reason for hiding this comment

Uh oh!

myasnikovdaniil commented Jun 10, 2026

Uh oh!

mattia-eleuteri commented Jun 10, 2026

Uh oh!

Aleksei Sviridkin (lexfrei) commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattia-eleuteri commented May 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 27, 2026 •

edited

Loading