Skip to content

fix(kubernetes): make the default md0 node group removable - #2753

Closed
mattia-eleuteri wants to merge 2 commits into
cozystack:mainfrom
mattia-eleuteri:feature/kubernetes-nodegroups-no-default-md0
Closed

mattia-eleuteri wants to merge 2 commits into
cozystack:mainfrom
mattia-eleuteri:feature/kubernetes-nodegroups-no-default-md0

Conversation

@mattia-eleuteri

@mattia-eleuteri mattia-eleuteri commented May 27, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Today the kubernetes chart ships a default nodeGroups.md0 in values.yaml. Because Helm deep-merges values, that md0 is re-added on top of any user-supplied nodeGroups, and since Kubernetes strips null values it can never be removed by configuration.

  • Chart: move the default md0 into a kubernetes.nodeGroups template helper, applied only when nodeGroups is empty. User-supplied node groups now fully replace the default — users can name and omit groups (including md0) freely. Used by both cluster.yaml and dashboard-resourcemap.yaml. Regenerated schema/README/-rd CRD.
  • Migration 44: makes md0 explicit on existing clusters that relied on the implicit default, to preserve their current node topology on the next reconcile. Idempotent; treats a missing CRD as a no-op; skips clusters that already define md0. Stamps version 45; migrations.targetVersion bumped to 45.

Notes for reviewers

  • This changes default behaviour: a cluster whose CR lists node groups without md0 will no longer get md0. On existing clusters the implicit md0 MachineDeployment is Helm-tracked, so without the migration it would be pruned (→ CAPI node deletion). Migration 44 pins it explicitly first (platform pre-upgrade hook ordering), so existing clusters are unaffected; operators can then remove md0 where unwanted.
  • ⚠️ Patching cluster CRs (migration 44) triggers a HelmRelease reconcile per cluster. Any reconcile of a tenant kubernetes app currently causes MetalLB L2 VIP re-announcement (brief edge outage, see Reconciling a tenant kubernetes app triggers MetalLB L2 VIP re-announcement and brief edge outage #2754) — until that is fixed, the migration should run during a maintenance window.

Release note

fix(kubernetes): the default `md0` node group is now applied only when no node groups are configured, so it can be removed or replaced. Migration 44 pins `md0` explicitly on existing clusters to preserve their topology.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Clarified node group docs: leaving nodeGroups empty provisions a single default md0 worker and user groups are not merged with that default.
  • Refactor

    • Internal node-group rendering updated to consistently use the documented default behavior.
  • Tests

    • CI/test fixtures updated to reflect explicit md0 defaults.
  • Chores

    • Migration and upgrade steps added to ensure existing installs receive the explicit md0 configuration and target version bump.

@coderabbitai

coderabbitai Bot commented May 27, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Chart values now default nodeGroups to empty; a new Helm helper emits either user-provided groups or a built-in md0 fallback. Templates consume the helper, values/schema/docs are updated, tests adjusted, ApplicationDefinition schema aligned, and a migration pins md0 on existing CRs.

Changes

Node Groups Default Provisioning

Layer / File(s) Summary
Helm defaults and schema contracts
packages/apps/kubernetes/values.yaml, packages/apps/kubernetes/values.schema.json, packages/apps/kubernetes/README.md, api/apps/v1alpha1/kubernetes/types.go
Chart nodeGroups default changed to {}, values/schema/docs/kubebuilder default updated to document that empty nodeGroups provisions a single default md0 and custom groups are not merged.
Node groups helper template
packages/apps/kubernetes/templates/_helpers.tpl
Adds kubernetes.nodeGroups helper that renders .Values.nodeGroups via toYaml when set, otherwise emits a built-in md0 node group with replica bounds, instanceType/disk defaults, roles: [ingress-nginx], empty resources/gpus, and empty kubelet.
Template loop integration
packages/apps/kubernetes/templates/cluster.yaml, packages/apps/kubernetes/templates/dashboard-resourcemap.yaml
Cluster and dashboard templates now iterate `include "kubernetes.nodeGroups" $
Tests and example values
packages/apps/kubernetes/tests/values-ci.yaml, packages/apps/kubernetes/tests/values/common.yaml, hack/e2e-apps/run-kubernetes.sh
CI/test values changed to include an explicit md0 node group where needed and set diskSize: 20Gi; E2E script now emits resources: {} for spec.nodeGroups.md0.
ApplicationDefinition schema alignment
packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
ApplicationDefinition openAPISchema updated to model nodeGroups as a map with per-node-group validation and keysOrder adjusted to general spec.nodeGroups entries.
Migration: pin implicit md0
packages/core/platform/images/migrations/migrations/44, packages/core/platform/values.yaml
Adds migration script to inject .spec.nodeGroups.md0 into existing kuberneteses when missing, with dry-run/idempotency/failure handling; advances migrations.targetVersion to 45.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • cozystack/cozystack#2780: Modifies packages/apps/kubernetes/templates/cluster.yaml’s node-group rendering path and relates to the same node-group object source.

Suggested labels

kind/api-change, area/platform

Suggested reviewers

  • lllamnyp
  • androndo
  • IvanHunters
  • sircthulhu
  • kvaps
  • lexfrei

Poem

🐰 From empty maps a group is made,
md0 hops forth to take the stage,
Helm sings helpers, templates trade,
Schemas tuned for the new page,
Cozy clusters center stage.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix(kubernetes): make the default md0 node group removable' clearly and concisely summarizes the main change: making the implicit default md0 node group optional/removable rather than permanently included.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Trivy (0.69.3)

Trivy execution failed: 2026-06-10T12:49:13Z FATAL Fatal error run error: fs scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: kubernetes scan error: fs filter error: fs filter error: walk error range error: stat smartylint.json: no such file or directory: range error: stat smartylint.json: no such file or directory

🔧 golangci-lint (2.12.2)

Command failed


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels May 27, 2026
@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

MetalLB reconcile/VIP-flap issue tracked in #2754 — relevant for migration 34's maintenance-window note.

@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

Validated on a live lab cluster with `cozyhr`:

  • Chart renders against a real cluster (instanceType lookup resolves) — rc=0.
  • Explicit nodeGroups → no forced md0: an app defining only worker-pool-1 renders no md0 resources (previously md0 was merged in). Footgun fixed.
  • Empty nodeGroups → md0 default: cozyhr show --plain with empty nodeGroups renders the default md0 MachineDeployment.
  • Migration 34: detection correctly targets CRs lacking md0; live patch pins md0 with the historical defaults while preserving existing groups; re-run is idempotent (skips); revert clean.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a limitation in the Kubernetes chart where the default 'md0' node group could not be removed due to Helm's deep-merging behavior. By shifting the default 'md0' definition into a template helper that only triggers when no node groups are configured, users gain full control over their node group topology. A migration script has been included to ensure existing clusters maintain their current state by explicitly pinning 'md0' before the change takes effect.

Highlights

  • Node Group Configuration: Moved the default 'md0' node group from values.yaml to a template helper, ensuring it is only applied when no user-defined node groups are present. This allows users to fully customize or remove the 'md0' group.
  • Migration 34: Introduced a migration script to explicitly pin the 'md0' node group on existing clusters, preventing accidental removal of the default node topology during the transition.
  • Documentation and Schema: Updated the README, values.yaml, and JSON schema to reflect the new node group configuration logic and provide clearer guidance for users.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors how the default md0 worker node group is handled in the Kubernetes application package. Instead of baking the default md0 group into values.yaml (which prevents users from fully overriding node groups due to Helm merging behavior), the default is now dynamically applied via a template helper only when nodeGroups is empty. A migration script (migrations/34) is added to explicitly pin the md0 group on existing clusters to preserve their topology. The review feedback highlights two issues in the migration script: a shell compatibility issue where pipefail is used with a #!/bin/sh shebang, and a fragile check using grep -q . to detect existing node groups which could be bypassed by null or <nil> outputs.

@@ -0,0 +1,47 @@
#!/bin/sh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The script uses set -euo pipefail on line 20, but the shebang is #!/bin/sh. The pipefail option is a bash extension and is not supported by standard POSIX shells (like dash, which is the default /bin/sh on Debian/Ubuntu). This will cause the migration script to fail immediately with an error like set: Illegal option -o pipefail at runtime.

Please change the shebang to #!/bin/bash to ensure the script runs correctly with pipefail enabled.

Suggested change
#!/bin/sh
#!/bin/bash

Comment on lines +33 to +34
if kubectl -n "$ns" get kuberneteses.apps.cozystack.io "$name" \
-o jsonpath='{.spec.nodeGroups.md0}' 2>/dev/null | grep -q .; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using grep -q . to check if md0 is defined can be fragile. Depending on the kubectl version and the exact state of the resource, a missing or null field might be output as null or <nil>, which grep -q . will incorrectly match as a valid value (since they are non-empty strings). This would cause the script to skip pinning md0 on clusters that actually need it.

Additionally, we can avoid spawning a subshell and grep process by capturing the output directly in a shell variable and performing a robust check.

Suggested change
if kubectl -n "$ns" get kuberneteses.apps.cozystack.io "$name" \
-o jsonpath='{.spec.nodeGroups.md0}' 2>/dev/null | grep -q .; then
res=$(kubectl -n "$ns" get kuberneteses.apps.cozystack.io "$name" -o jsonpath='{.spec.nodeGroups.md0}' 2>/dev/null)
if [ -n "$res" ] && [ "$res" != "null" ] && [ "$res" != "<nil>" ]; then

@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

Important: existing-cluster prune behaviour (and why migration 34 is required)

On existing clusters the md0 MachineDeployment is already a Helm-tracked object in the release manifest (verified on a live cluster: the current release secret's manifest contains …-md0). So upgrading to this chart, which no longer renders md0 for clusters that define their own node groups, would make Helm prune the live md0 MachineDeployment → CAPI deletes the MachineSet/Machine → the node is drained. A render-only check (cozyhr show) does not surface this — it only shows the desired set.

Migration 34 prevents the prune by pinning md0 into the CR before the new chart reconciles, so md0 stays in the rendered manifest. Verified on a lab cluster:

  • CR with only worker-pool-1 + new chart → md0 absent from render (would prune the live md0).
  • After migration 34 pins md0 → new chart renders both md0 and worker-pool-1 → no prune.

Ordering: migration 34 runs via the platform chart's helm.sh/hook: pre-upgrade,pre-install (weight 1), i.e. before the platform release applies. This is the intended "migrate data before new code" sequence. To confirm end-to-end, a full old→new upgrade should be run on a lab cluster that has a live md0 MachineDeployment, verifying md0 is preserved (not pruned).

@mattia-eleuteri
mattia-eleuteri marked this pull request as draft May 27, 2026 12:47

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/core/platform/images/migrations/migrations/34`:
- Around line 27-29: The kubectl listing command (kubectl get
kuberneteses.apps.cozystack.io -A ...) can fail under set -euo pipefail when the
CRD is absent, causing the migration to abort; change the logic to capture the
kubectl output safely (e.g. output=$(kubectl get kuberneteses.apps.cozystack.io
-A -o jsonpath=... 2>/dev/null || true)), then check if output is empty and
treat that as a successful no-op (exit/return success) otherwise pipe the output
into the existing while read -r key loop to continue processing; reference the
existing kubectl get ... command and the while read -r key loop when making the
change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: eb08a3d3-c7d1-4985-b9c9-0f9c6a324efe

📥 Commits

Reviewing files that changed from the base of the PR and between 72c4eb9 and 7ad9da0.

📒 Files selected for processing (8)
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/templates/_helpers.tpl
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/templates/dashboard-resourcemap.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/core/platform/images/migrations/migrations/34
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml

Comment on lines +27 to +29
kubectl get kuberneteses.apps.cozystack.io -A \
-o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' \
| while read -r key; do

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Handle missing CRD/no-instance list as a successful no-op.

With set -euo pipefail, kubectl get kuberneteses.apps.cozystack.io -A fails hard when the CRD is absent, so the migration exits instead of completing successfully.

💡 Suggested fix
-kubectl get kuberneteses.apps.cozystack.io -A \
-  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' \
-  | while read -r key; do
+kubectl get kuberneteses.apps.cozystack.io -A \
+  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' 2>/dev/null \
+  | while read -r key; do
     [ -z "$key" ] && continue
     ns="${key%%/*}"
     name="${key##*/}"
@@
-  done
+  done || true

Based on learnings: "if a kubectl list returns no results (CRD not present or no instances), the migration should complete successfully without requiring an explicit CRD existence check."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/platform/images/migrations/migrations/34` around lines 27 - 29,
The kubectl listing command (kubectl get kuberneteses.apps.cozystack.io -A ...)
can fail under set -euo pipefail when the CRD is absent, causing the migration
to abort; change the logic to capture the kubectl output safely (e.g.
output=$(kubectl get kuberneteses.apps.cozystack.io -A -o jsonpath=...
2>/dev/null || true)), then check if output is empty and treat that as a
successful no-op (exit/return success) otherwise pipe the output into the
existing while read -r key loop to continue processing; reference the existing
kubectl get ... command and the while read -r key loop when making the change.

@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

Deterministic manifest-level proof of the prune (and mitigation)

Verified against a live cluster's Helm release for an app whose CR defines only worker-pool-1 (no md0):

Source md0 objects
Current Helm release manifest (what Helm tracks) 13 (present)
This PR's render, CR unchanged 0
This PR's render, md0 pinned in CR (post-migration-34) 13

helm upgrade deletes objects present in the previous release manifest but absent from the new render. So on existing clusters this PR prunes the live md0 MachineDeployment (→ CAPI MachineSet/Machine/node deletion) unless migration 34 pins md0 first, which it does.

Note: a full live upgrade (watching the node) still needs a cluster that can provision healthy tenant clusters; this is best confirmed on staging in a maintenance window. An alternative design worth discussing: have the chart keep md0 when a live md0 MachineDeployment is detected via lookup, removing the dependency on migration timing entirely.

@mattia-eleuteri
mattia-eleuteri force-pushed the feature/kubernetes-nodegroups-no-default-md0 branch from 7ad9da0 to feccf3a Compare May 27, 2026 14:57
@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

Rebased onto latest `main` and addressed review feedback:

  • Migration renumbered 34 → 43 (34 is now upstream's RabbitMQ migration); bumped `migrations.targetVersion` to 44.
  • Gemini (@gemini-code-assist) (shebang/pipefail): migration now uses `#!/bin/bash` (consistent with migration 41).
  • Gemini (@gemini-code-assist) (fragile `grep -q .`): replaced with a `jq`-based presence check that only adds `md0` when `.spec.nodeGroups` is an object lacking `md0`.
  • CodeRabbit (@coderabbitai) (CRD-absent hard fail): migration now treats a missing CRD as a successful no-op (mirrors migration 41's `list_objects` guard), and is best-effort (logs failures, leaves the version stamp for retry).
  • Adapted to the `ephemeralStorage` → `diskSize` rename: the default `md0` (helper + migration) now uses `diskSize`/`storageClass`/`kubelet`.
  • Regenerated schema/CRD/README/types with cozyvalues-gen v1.5.0; all 124 chart unit tests pass.

@coderabbitai

coderabbitai Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find migration 43 file
fd -t f '43' packages/core/platform/images/migrations/migrations/
echo "---"
cat packages/core/platform/images/migrations/migrations/43 2>/dev/null || echo "not found at that path"
echo "---"
# Also check if there's a different path structure
find packages/core/platform/images/migrations -type f | sort

Length of output: 6870



@mattia-eleuteri Thanks for the thorough update and for addressing all the review feedback. Verified migration 43 directly — everything looks solid:

  • CRD-absent guard (list_objects): correctly catches "the server doesn't have a resource type", "no matches for kind", and "NotFound", returning 0 (no-op) — my concern is fully addressed.
  • jq presence check: (.spec.nodeGroups | has("md0") | not) is clean and unambiguous.
  • diskSize/storageClass/kubelet in MD0_JSON: matches the renamed fields.
  • Best-effort + retry: FAILURES counter, version stamp held at 43 on partial failure so the next platform upgrade retries — good resilience pattern.
  • Dry-run support via MIGRATION_DRY_RUN: nice addition for pre-flight validation.

The migration-before-chart ordering via helm.sh/hook: pre-upgrade,pre-install (weight 1) is the right sequencing. The remaining item you noted — a full old→new live upgrade on a staging cluster with a live md0 MachineDeployment — is the sensible final gate before merging.

🐇 ✓

[resolve]

@mattia-eleuteri
mattia-eleuteri force-pushed the feature/kubernetes-nodegroups-no-default-md0 branch from feccf3a to 1e21392 Compare May 28, 2026 07:01
@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

The previous E2E run failed on the 3 tenant-Kubernetes tests (`kubernetes-latest`, `kubernetes-previous`, `ouroboros`). Root cause was the same pattern as the unit-test fixtures:

```
Helm install failed […]: values don't meet the specifications of the schema:

  • at '/nodeGroups/md0': missing property 'resources'
    ```

The E2E test CRs (`hack/e2e-apps/run-kubernetes.sh`, `hack/e2e-apps/ouroboros.bats`) defined `md0` without `resources` (and ouroboros also without `gpus`). On main, the chart's values.yaml default `md0` supplied those required fields via Helm values merge; removing the default in this PR exposed that the test CRs relied on it. Updated both fixtures to declare those fields explicitly. Re-running CI.

@mattia-eleuteri
mattia-eleuteri force-pushed the feature/kubernetes-nodegroups-no-default-md0 branch from 1e21392 to 5462dea Compare May 28, 2026 09:34
@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

Re-rebased onto latest main (PRs #2728 / #2759 / #2726 / #2743 landed). ouroboros.bats was folded into kubernetes-latest.bats by #2728 and removed — dropped my earlier change to that file. run-kubernetes.sh already includes resources: {} in upstream after #2728, so the test-CR fix from my last push is now redundant there (auto-merge kept the unified content). Migration is now numbered 43, targetVersion 44. Local unit tests still pass (124/124). Waiting on the re-triggered CI.

@mattia-eleuteri
mattia-eleuteri marked this pull request as ready for review May 28, 2026 12:26

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors how the default md0 worker node group is handled in the Kubernetes application chart. Instead of defining the default md0 group in values.yaml (which interferes with Helm values merging when users specify custom node groups), the default is now dynamically applied in the Helm templates via a helper when nodeGroups is left empty. To ensure backward compatibility and prevent existing tenant clusters from losing their md0 node group, a new platform migration script (43) has been added to explicitly pin the md0 configuration on existing clusters, and the platform's target migration version has been bumped to 44. There are no review comments, and I have no feedback to provide.

@coderabbitai

coderabbitai Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Actionable comments posted: 0

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

This has gone stale against main (~190 commits). Two real steps to rebase, beyond the usual regenerate:

1. Migration number collision. main now ships its own migration 43 (seaweedfs-db adoption), so yours has to move to 44:

  • rename packages/core/platform/images/migrations/migrations/43 → 44 (keep the executable bit, chmod +x)
  • inside that file: header # Migration 43 --> 44 → # Migration 44 --> 45; the two "version stamp at 43" comments → 44; and the final --from-literal version=44 → version=45
  • bump packages/core/platform/values.yaml targetVersion: 44 → 45

The runner loops seq CURRENT_VERSION (TARGET_VERSION-1) and each migration self-stamps its target, so file 44 migrates 44 → 45 and targetVersion must be 45.

2. Generated cozyrds conflict — after rebasing, cd packages/apps/kubernetes && make generate and commit the regenerated files.

Minor: the commit title says "add migration 43" — worth updating to 44 once renumbered.

… empty

The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.

Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.

Signed-off-by: mattia-eleuteri <[email protected]>
After the kubernetes chart stops force-merging a default md0 node group,
clusters that relied on the implicit md0 (defined other nodeGroups without
md0) would have their live md0 MachineDeployment pruned by Helm on the next
reconcile. Migration 44 makes md0 explicit on those clusters to preserve the
current node topology; it is idempotent, treats a missing CRD as a no-op, and
skips clusters that already define md0.

Signed-off-by: mattia-eleuteri <[email protected]>
@mattia-eleuteri
mattia-eleuteri force-pushed the feature/kubernetes-nodegroups-no-default-md0 branch from 5462dea to d73d1d9 Compare June 10, 2026 12:47
@mattia-eleuteri

Copy link
Copy Markdown
Collaborator Author

myasnikovdaniil done as instructed: rebased onto latest main; migration renumbered 43 → 44 (main's 43 = seaweedfs-db adoption kept as-is), header now 44 --> 45, version stamp 45, targetVersion: 45, executable bit preserved; commit title updated to "migration 44"; cozyrds regenerated with cozyvalues-gen v1.6.0. 129 chart unit tests pass.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/core/platform/images/migrations/migrations/44`:
- Around line 97-99: The script currently exits with code 0 when FAILURES > 0
which signals success; change the exit behavior so that when the FAILURES
variable is greater than zero the script exits with a non-zero status (e.g.,
exit 1) instead of exit 0 to ensure run-migrations.sh treats the migration as
failed; update the branch that prints "WARN: $FAILURES patch(es) failed; leaving
CURRENT_VERSION at 44 for retry on next upgrade." to call exit 1 while keeping
the warning message and leaving CURRENT_VERSION untouched.
- Around line 102-104: The migration script hard-codes the namespace
"cozy-system" when creating the cozystack-version ConfigMap, which breaks
non-default installs; change the kubectl command to use the release namespace
used elsewhere (the same value passed as NAMESPACE / .Release.Namespace) so the
ConfigMap is created/updated in the release namespace (cozystack-version) rather
than always in cozy-system; update the command invocation that references
cozystack-version to substitute the release namespace variable
(.Release.Namespace or the NAMESPACE env var) instead of the literal
"cozy-system".
- Around line 55-56: The current line that runs obj=$(kubectl get
kuberneteses.apps.cozystack.io ... 2>/dev/null || true) swallows all kubectl
errors and treats them as SKIPPED; change it to capture kubectl's exit status
and stderr and only increment SKIPPED when the error is an actual NotFound
(e.g., "NotFound" / 404 from the API). Specifically, replace the unconditional
2>/dev/null || true pattern around the kubectl get (the obj assignment) with
logic that records the command's exit code, inspects stderr/output for a
NotFound indication, and: if rc==0 proceed using obj, if error indicates
NotFound increment SKIPPED and continue, otherwise surface/log the error and
exit non‑zero so transient API errors don't get treated as successful skips.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: be0a9fdf-949a-449a-b2fa-44d5feccb962

📥 Commits

Reviewing files that changed from the base of the PR and between 5462dea and d73d1d9.

📒 Files selected for processing (12)
  • api/apps/v1alpha1/kubernetes/types.go
  • hack/e2e-apps/run-kubernetes.sh
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/templates/_helpers.tpl
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/templates/dashboard-resourcemap.yaml
  • packages/apps/kubernetes/tests/values-ci.yaml
  • packages/apps/kubernetes/tests/values/common.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/core/platform/images/migrations/migrations/44
  • packages/core/platform/values.yaml
✅ Files skipped from review due to trivial changes (1)
  • packages/core/platform/values.yaml
🚧 Files skipped from review as they are similar to previous changes (4)
  • packages/apps/kubernetes/templates/cluster.yaml
  • hack/e2e-apps/run-kubernetes.sh
  • api/apps/v1alpha1/kubernetes/types.go
  • packages/apps/kubernetes/README.md

Comment on lines +55 to +56
obj=$(kubectl get kuberneteses.apps.cozystack.io --namespace "$ns" "$name" --output json 2>/dev/null || true)
if [ -z "$obj" ]; then SKIPPED=$((SKIPPED + 1)); return 0; fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't collapse kubectl get failures into a skip.

If this per-object read hits a transient API error, 2>/dev/null || true turns it into an empty string, increments SKIPPED, and lets the migration keep going. The script can then still stamp version 45 even though that cluster never got md0, which reintroduces the prune risk this migration is meant to prevent.

Suggested fix
-  obj=$(kubectl get kuberneteses.apps.cozystack.io --namespace "$ns" "$name" --output json 2>/dev/null || true)
-  if [ -z "$obj" ]; then SKIPPED=$((SKIPPED + 1)); return 0; fi
+  if ! obj=$(kubectl get kuberneteses.apps.cozystack.io --namespace "$ns" "$name" --output json 2>&1); then
+    case "$obj" in
+      *"NotFound"*)
+        SKIPPED=$((SKIPPED + 1))
+        return 0
+        ;;
+    esac
+    echo "WARN: failed to read $ns/$name: $obj" >&2
+    FAILURES=$((FAILURES + 1))
+    return 0
+  fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
obj=$(kubectl get kuberneteses.apps.cozystack.io --namespace "$ns" "$name" --output json 2>/dev/null || true)
if [ -z "$obj" ]; then SKIPPED=$((SKIPPED + 1)); return 0; fi
if ! obj=$(kubectl get kuberneteses.apps.cozystack.io --namespace "$ns" "$name" --output json 2>&1); then
case "$obj" in
*"NotFound"*)
SKIPPED=$((SKIPPED + 1))
return 0
;;
esac
echo "WARN: failed to read $ns/$name: $obj" >&2
FAILURES=$((FAILURES + 1))
return 0
fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/platform/images/migrations/migrations/44` around lines 55 - 56,
The current line that runs obj=$(kubectl get kuberneteses.apps.cozystack.io ...
2>/dev/null || true) swallows all kubectl errors and treats them as SKIPPED;
change it to capture kubectl's exit status and stderr and only increment SKIPPED
when the error is an actual NotFound (e.g., "NotFound" / 404 from the API).
Specifically, replace the unconditional 2>/dev/null || true pattern around the
kubectl get (the obj assignment) with logic that records the command's exit
code, inspects stderr/output for a NotFound indication, and: if rc==0 proceed
using obj, if error indicates NotFound increment SKIPPED and continue, otherwise
surface/log the error and exit non‑zero so transient API errors don't get
treated as successful skips.

Comment on lines +97 to +99
if [ "$FAILURES" -gt 0 ]; then
echo "WARN: $FAILURES patch(es) failed; leaving CURRENT_VERSION at 44 for retry on next upgrade."
exit 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Return non-zero when any patch fails.

run-migrations.sh treats exit code 0 as “migration completed” and immediately continues to later migration numbers in the same job. Once a future release sets targetVersion > 45, this script will allow migration 45+ to run even though 44 explicitly did not finish.

Suggested fix
 if [ "$FAILURES" -gt 0 ]; then
   echo "WARN: $FAILURES patch(es) failed; leaving CURRENT_VERSION at 44 for retry on next upgrade."
-  exit 0
+  exit 1
 fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [ "$FAILURES" -gt 0 ]; then
echo "WARN: $FAILURES patch(es) failed; leaving CURRENT_VERSION at 44 for retry on next upgrade."
exit 0
if [ "$FAILURES" -gt 0 ]; then
echo "WARN: $FAILURES patch(es) failed; leaving CURRENT_VERSION at 44 for retry on next upgrade."
exit 1
fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/platform/images/migrations/migrations/44` around lines 97 - 99,
The script currently exits with code 0 when FAILURES > 0 which signals success;
change the exit behavior so that when the FAILURES variable is greater than zero
the script exits with a non-zero status (e.g., exit 1) instead of exit 0 to
ensure run-migrations.sh treats the migration as failed; update the branch that
prints "WARN: $FAILURES patch(es) failed; leaving CURRENT_VERSION at 44 for
retry on next upgrade." to call exit 1 while keeping the warning message and
leaving CURRENT_VERSION untouched.

Comment on lines +102 to +104
kubectl create configmap --namespace cozy-system cozystack-version \
--from-literal version=45 --dry-run=client --output yaml \
| kubectl apply --filename -

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use the release namespace for cozystack-version.

The hook and the template both read/write this ConfigMap in .Release.Namespace and pass that value in NAMESPACE. Hard-coding cozy-system here updates the wrong object for non-default installs, so the real release namespace stays at version 44 and reruns migrations forever.

Suggested fix
-kubectl create configmap --namespace cozy-system cozystack-version \
+kubectl create configmap --namespace "${NAMESPACE:?NAMESPACE is required}" cozystack-version \
   --from-literal version=45 --dry-run=client --output yaml \
   | kubectl apply --filename -
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
kubectl create configmap --namespace cozy-system cozystack-version \
--from-literal version=45 --dry-run=client --output yaml \
| kubectl apply --filename -
kubectl create configmap --namespace "${NAMESPACE:?NAMESPACE is required}" cozystack-version \
--from-literal version=45 --dry-run=client --output yaml \
| kubectl apply --filename -
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/platform/images/migrations/migrations/44` around lines 102 -
104, The migration script hard-codes the namespace "cozy-system" when creating
the cozystack-version ConfigMap, which breaks non-default installs; change the
kubectl command to use the release namespace used elsewhere (the same value
passed as NAMESPACE / .Release.Namespace) so the ConfigMap is created/updated in
the release namespace (cozystack-version) rather than always in cozy-system;
update the command invocation that references cozystack-version to substitute
the release namespace variable (.Release.Namespace or the NAMESPACE env var)
instead of the literal "cozy-system".

Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Jun 30, 2026
Same-repo mirror of #2753 by @mattia-eleuteri, opened so the `Build` CI
job can run. Fork PRs skip the OCIR registry login (`if:
!github.event.pull_request.head.repo.fork` in `pull-requests.yaml`) and
so can't push the per-PR images the build needs — meaning a fork PR can
never pass `Build`, regardless of the change. The commits here are
unchanged and authored by @mattia-eleuteri; full description and
discussion in #2753.

This includes migration **44** (renumbered from 43 after the
seaweedfs-db migration landed on `main`) and bumps `targetVersion` to
**45**.

Supersedes #2753.

```release-note
fix(kubernetes): the default `md0` node group is now applied only when no node groups are configured, so it can be removed or replaced. Migration 44 pins `md0` explicitly on existing clusters to preserve their topology.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Empty `nodeGroups` settings now automatically provision a default
`md0` worker group.
* Existing environments are updated to keep this default group available
during upgrades.

* **Bug Fixes**
* User-defined node groups now fully replace the default instead of
being merged with it.
* Improved handling of node-group resources and permissions in
Kubernetes app rendering.

* **Documentation**
* Updated Kubernetes app docs and schema details to reflect the new
default behavior.

* **Tests**
* Added coverage for default and custom node-group configurations, plus
migration behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@lexfrei

Copy link
Copy Markdown
Contributor

Closing: this is effectively already in main. The default md0 node group is now applied only when nodeGroups is empty, via the nodeGroups: {} default plus the kubernetes.nodeGroups helper — the same mechanism proposed here. The migration slot used by this PR (45) is also outdated: migrations.targetVersion has since advanced to 48. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants