Skip to content

[kubernetes] Fixed k8s<1.32 creation - #2209

Merged
Andrei Kvapil (kvaps) merged 1 commit into
mainfrom
fix/tenant-k8s
Mar 13, 2026
Merged

Andrei Kvapil (kvaps) merged 1 commit into
mainfrom
fix/tenant-k8s

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Mar 12, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

This PR adds version specific ubuntu base images to fix errors when base image has new deb packages of kubeadm and kubelet installed, but at runtime it was downgraded by replacing just binaries. Now update by replacing binaries works as intended - latest patch version of minor version used.

Core issue was in kubeadm<1.32 expecting conntrack binary in its preflight checks but it was not found. It happened because kubelet deb package dropped conntrack dependency since 1.32 (actually it absent in 1.31.14 too).
So now status of supported tenant k8s versions is:

  • 1.30 - works because kubelet package provided conntrack, also conntrack preflight check ignored (see 1.31).
  • 1.31 - works because conntrack preflight check ignored (for some reason kubelet 1.31.14 did't provide conntrack dependency, unlike 1.31.13 did).
  • >=1.32 - works because conntrack preflight check removed from kubeadm init entirely.

Conntrack preflight check ignoring is legit for tenant kubernetes clusters because until 1.32 it was used in kube-proxy but cozystack k8s approach doesn't use kube-proxy (replaced with cilium).

Issue with conntrack may be mitigated with only ignorePreflightErrors, but I think proper base image build will help to avoid similar bugs in future.

Release note

[kubernetes] Fixed tenant k8s older than 1.32 creation by adding version specific ubuntu base images

Summary by CodeRabbit

Release Notes

  • New Features
    • Added multi-version Kubernetes support with version-specific container images.
    • Enhanced compatibility with newer Kubernetes releases, including version 1.31.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the summary. You can try again by commenting /gemini summary.

@dosubot dosubot Bot added size/M This PR changes 30-99 lines, ignoring generated files kind/bug Categorizes issue or PR as related to a bug labels Mar 12, 2026
@coderabbitai

coderabbitai Bot commented Mar 12, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The pull request transitions from a single Kubernetes version build to a multi-version approach. The Makefile now reads versions from a YAML configuration file and iterates through multiple Kubernetes versions (1.30–1.35), generating per-version image tags and metadata files sequentially instead of maintaining a single unified build.

Changes

Cohort / File(s) Summary
Build System Configuration
packages/apps/kubernetes/Makefile
Replaces KUBERNETES_VERSION with KUBERNETES_VERSIONS by reading versions.yaml. Refactors image-ubuntu-container-disk to loop over versions, building per-version images with corresponding tags and metadata files.
Container Image Tags
packages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tag, ubuntu-container-disk-v1.31.tag, ubuntu-container-disk-v1.32.tag, ubuntu-container-disk-v1.33.tag, ubuntu-container-disk-v1.34.tag
Added new per-version container image references with SHA256 digests for Kubernetes versions 1.30 through 1.35.
Legacy Image Tag
packages/apps/kubernetes/images/ubuntu-container-disk.tag
Removed single-version image tag file, replaced by versioned variants.
Cluster Configuration
packages/apps/kubernetes/templates/cluster.yaml
Updated container disk image reference to use versioned tag injection. Added ignorePreflightErrors configuration for kubelet compatibility with Kubernetes 1.31.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A rabbit hops through versions bright,
Multi-builds now shine their light!
From one to many, tags align,
Each Kubernetes version in line divine!

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fixed k8s<1.32 creation' directly summarizes the main objective of the PR—addressing Kubernetes version compatibility issues below 1.32 by introducing version-specific base images.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/tenant-k8s

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
packages/apps/kubernetes/Makefile (1)

1-1: Update hack/update-versions.sh for KUBERNETES_VERSIONS.

make update still calls a script that rewrites KUBERNETES_VERSION in this file, so after this change that branch only emits a warning. Cleaning up that stale step will keep the update workflow consistent with files/versions.yaml.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kubernetes/Makefile` at line 1, The Makefile now derives
KUBERNETES_VERSIONS from files/versions.yaml (KUBERNETES_VERSIONS = $(shell awk
-F'"' '{print $$2}' files/versions.yaml)) but the update workflow still runs
hack/update-versions.sh which rewrites a stale KUBERNETES_VERSION in this file;
update hack/update-versions.sh (invoked by the make update target) to stop
modifying the Makefile (remove or change the branch that writes
KUBERNETES_VERSION) so it either updates files/versions.yaml only or emits a
consistent warning, ensuring no stale rewrite of KUBERNETES_VERSIONS occurs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/apps/kubernetes/Makefile`:
- Around line 20-23: The build currently uses --build-arg
KUBERNETES_VERSION=$(ver) which only carries the minor tag; change the Makefile
to also pass the exact mapped patch from versions.yaml (e.g., add --build-arg
KUBERNETES_FULL_VERSION=$(call settag,$(ver)-$(TAG)) or a new variable that
resolves the exact patch) so the image build is pinned to the same patch applied
by update-k8s.sh; update the Dockerfile to accept ARG KUBERNETES_FULL_VERSION
and use that value when installing/pinning kubelet and kubeadm package versions
instead of the minor-only KUBERNETES_VERSION.
- Around line 18-30: The foreach loop over KUBERNETES_VERSIONS masks per-version
failures because iterations are separated with ';' so a failed build can be
followed by successful ones and the target exits 0; update the loop body in the
Makefile so each iteration chain uses '&&' instead of ';' (affecting the docker
buildx build, echo and rm steps for images/ubuntu-container-disk and settag
usage) and append '&& true' after the closing ')' of the foreach to ensure the
overall make recipe correctly propagates any per-version failures.

---

Nitpick comments:
In `@packages/apps/kubernetes/Makefile`:
- Line 1: The Makefile now derives KUBERNETES_VERSIONS from files/versions.yaml
(KUBERNETES_VERSIONS = $(shell awk -F'"' '{print $$2}' files/versions.yaml)) but
the update workflow still runs hack/update-versions.sh which rewrites a stale
KUBERNETES_VERSION in this file; update hack/update-versions.sh (invoked by the
make update target) to stop modifying the Makefile (remove or change the branch
that writes KUBERNETES_VERSION) so it either updates files/versions.yaml only or
emits a consistent warning, ensuring no stale rewrite of KUBERNETES_VERSIONS
occurs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 33c3aa93-796b-437b-8ef1-6ae7c7b7ec28

📥 Commits

Reviewing files that changed from the base of the PR and between bf472fb and f82f13b.

📒 Files selected for processing (9)
  • packages/apps/kubernetes/Makefile
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.30.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.31.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.32.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.33.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.34.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk-v1.35.tag
  • packages/apps/kubernetes/images/ubuntu-container-disk.tag
  • packages/apps/kubernetes/templates/cluster.yaml
💤 Files with no reviewable changes (1)
  • packages/apps/kubernetes/images/ubuntu-container-disk.tag

Comment on lines +18 to +30
$(foreach ver,$(KUBERNETES_VERSIONS), \
docker buildx build images/ubuntu-container-disk \
--build-arg KUBERNETES_VERSION=$(ver) \
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)-$(TAG)) \
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
--cache-to type=inline \
--metadata-file images/ubuntu-container-disk-$(ver).json \
$(BUILDX_ARGS) && \
echo "$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver))@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
> images/ubuntu-container-disk-$(ver).tag && \
rm -f images/ubuntu-container-disk-$(ver).json; \
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== image-ubuntu-container-disk recipe =="
sed -n '17,30p' packages/apps/kubernetes/Makefile

echo
echo "== Shell behavior with ';' after a failed && chain =="
bash -lc 'false && echo skipped && echo skipped; true'
echo "exit=$?"

Repository: cozystack/cozystack

Length of output: 1053


Fix error handling in foreach loop to prevent masking per-version build failures.

The ; terminator at the end of each iteration breaks the error chain. If any version's build fails, the loop continues to the next version, and the target exits 0 if the last iteration succeeds—even though earlier versions failed partially. Change the terminator from ; to && and add true after the closing paren to preserve error propagation across all iterations.

🛠️ Suggested change
 image-ubuntu-container-disk:
 	$(foreach ver,$(KUBERNETES_VERSIONS), \
 		docker buildx build images/ubuntu-container-disk \
 			--build-arg KUBERNETES_VERSION=$(ver) \
 			--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
 			--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)-$(TAG)) \
 			--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
 			--cache-to type=inline \
 			--metadata-file images/ubuntu-container-disk-$(ver).json \
 			$(BUILDX_ARGS) && \
 		echo "$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver))@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
 			> images/ubuntu-container-disk-$(ver).tag && \
-		rm -f images/ubuntu-container-disk-$(ver).json; \
-	)
+		rm -f images/ubuntu-container-disk-$(ver).json && \
+	) true
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kubernetes/Makefile` around lines 18 - 30, The foreach loop
over KUBERNETES_VERSIONS masks per-version failures because iterations are
separated with ';' so a failed build can be followed by successful ones and the
target exits 0; update the loop body in the Makefile so each iteration chain
uses '&&' instead of ';' (affecting the docker buildx build, echo and rm steps
for images/ubuntu-container-disk and settag usage) and append '&& true' after
the closing ')' of the foreach to ensure the overall make recipe correctly
propagates any per-version failures.

Comment on lines +20 to +23
--build-arg KUBERNETES_VERSION=$(ver) \
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)-$(TAG)) \
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Minor -> patch mapping =="
sed -n '1,20p' packages/apps/kubernetes/files/versions.yaml

echo
echo "== Build args passed from the Makefile =="
rg -n -C2 'image-ubuntu-container-disk|--build-arg KUBERNETES_VERSION|metadata-file images/ubuntu-container-disk' packages/apps/kubernetes/Makefile

echo
echo "== Kubernetes-related ARGs/usages in the Dockerfile =="
rg -n -C2 '^(ARG|ENV)\s+KUBERNETES|KUBERNETES_VERSION|kubelet|kubeadm' packages/apps/kubernetes/images/ubuntu-container-disk/Dockerfile

echo
echo "== Runtime patch pin =="
rg -n -C1 'KUBELET_VERSION=.*kubernetes\.versionMap' packages/apps/kubernetes/templates/cluster.yaml

Repository: cozystack/cozystack

Length of output: 2572


Thread the exact patch version from versions.yaml into the image build.

The Makefile currently passes only the minor version (e.g., v1.30) to the Docker build, while the exact patch (e.g., v1.30.14) is applied later at runtime via update-k8s.sh. This disconnects the build from the intended patch level. Since versions.yaml intentionally caps patches to Kamaji's supported boundary, the image build should explicitly pin the exact patch to guarantee consistency between build-time and runtime versions.

Show current build invocation
--build-arg KUBERNETES_VERSION=$(ver) \

Pass the mapped exact patch to the Dockerfile (e.g., via a second build argument), and use it to pin kubelet/kubeadm versions in the package installation step.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kubernetes/Makefile` around lines 20 - 23, The build currently
uses --build-arg KUBERNETES_VERSION=$(ver) which only carries the minor tag;
change the Makefile to also pass the exact mapped patch from versions.yaml
(e.g., add --build-arg KUBERNETES_FULL_VERSION=$(call settag,$(ver)-$(TAG)) or a
new variable that resolves the exact patch) so the image build is pinned to the
same patch applied by update-k8s.sh; update the Dockerfile to accept ARG
KUBERNETES_FULL_VERSION and use that value when installing/pinning kubelet and
kubeadm package versions instead of the minor-only KUBERNETES_VERSION.

@sircthulhu Kirill Ilin (sircthulhu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Mar 12, 2026
@kvaps
Andrei Kvapil (kvaps) merged commit 39df525 into main Mar 13, 2026
13 checks passed
@kvaps
Andrei Kvapil (kvaps) deleted the fix/tenant-k8s branch March 13, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug lgtm This PR has been approved by a maintainer size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants