Skip to content

fix(kamaji): close Go stdlib and module CVEs in the shipped binary - #3033

Merged
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
fix/kamaji-rebuild-stdlib-cve
Jun 25, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
fix/kamaji-rebuild-stdlib-cve

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

What

Rebuild the kamaji image and force the fixed versions of three Go modules whose advisories are compiled into the manager binary. Two build changes:

  • Advance the build source from 26.3.5-edge to 26.3.6-edge, recompiling on the current golang:1.26 toolchain (1.26.4) to close the Go stdlib advisories.
  • After applying the source patches and before go build, run go get golang.org/x/[email protected] golang.org/x/[email protected] go.opentelemetry.io/otel/[email protected] && go mod tidy to pull the patched module versions into the build.

Why

The pinned image is flagged for a set of CVEs that fall into three groups:

  • 16 OS-package findings (libpython, libexpat, gnutls, krb5, sqlite, curl, libgcrypt, libcap, nghttp2). The base is gcr.io/distroless/static:nonroot, which ships no OS packages, so these do not apply to the image — false positives.
  • 1 stdlib finding: the binary was compiled with a go1.26 patch older than 1.26.4. The 26.3.6-edge rebuild on the current toolchain closes it (crypto/x509 CVE-2026-27145, mime CVE-2026-42504, net/textproto CVE-2026-42507).
  • Go-module advisories that, unlike the OS-package noise, are genuinely present in the compiled binary. 26.3.6-edge pins golang.org/x/net v0.51.0, golang.org/x/crypto v0.49.0 and go.opentelemetry.io/otel/sdk v1.41.0, all carrying published advisories:

A source-version bump cannot close the module advisories: no kamaji edge tag up to 26.6.4-edge ships the fixed module versions (it still pins x/net 0.53.0, x/crypto 0.51.0, otel/sdk 1.41.0), and the tags that advance the k8s line to 1.36 regress tenant control-plane RBAC. Forcing the modules during the build is the minimal correct lever, and the overlay already triggers a fresh compile, so it subsumes the stdlib rebuild.

Verification

Built the image source end to end: extracted 26.3.6-edge, applied the patch, ran the overlay (go get + go mod tidy, clean), and compiled with CGO_ENABLED=0 go build -a -o kamaji main.go. The build succeeds and go version -m on the binary embeds golang.org/x/net v0.55.0, golang.org/x/crypto v0.52.0 and the go.opentelemetry.io/otel set at v1.43.0. The source has no vendor/ directory, so go get + go mod tidy is sufficient. Advisory IDs and fixed boundaries verified against the OSV database.

After this change the shipped binary carries no Go advisory with an available fix; the only residual scanner entries are in k8s.io/kubernetes with no upstream fix, which no version bump can close.

Closes #2885

Summary by CodeRabbit

  • Chores
    • Updated the Kamaji image version to a newer patch release.
    • Refreshed several bundled Go dependencies to newer pinned versions before building.

The shipped kamaji binary was compiled with go1.26.3, which trivy flags
for the standard-library CVEs fixed in go1.26.4: crypto/x509
CVE-2026-27145, mime CVE-2026-42504, and net/textproto CVE-2026-42507.
The image base is distroless/static, so the remaining OS-package CVEs
from the scan do not apply to it.

Advance the build source from 26.3.5-edge to 26.3.6-edge, the next
patch on the same edge line (identical k8s.io/api v0.35.0 and
controller-runtime v0.22.4, staying clear of the k8s 1.36 line that
regresses tenant control-plane RBAC). The version change busts the
build-layer cache so the release rebuild compiles against the current
golang:1.26 toolchain, which resolves to 1.26.4 and closes the stdlib
CVEs.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@coderabbitai

coderabbitai Bot commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4019d2c8-0137-4d00-8b86-46a517f7ec80

📥 Commits

Reviewing files that changed from the base of the PR and between 29bf56f and 13ca721.

📒 Files selected for processing (1)
  • packages/system/kamaji/images/kamaji/Dockerfile

📝 Walkthrough

Walkthrough

Updates the Kamaji image build to use 26.3.6-edge and adds a build-time step that pins specific Go module versions before compiling the manager binary.

Changes

Kamaji image build update

Layer / File(s) Summary
Version tag bump
packages/system/kamaji/images/kamaji/Dockerfile
Updates the builder-stage VERSION argument from 26.3.5-edge to 26.3.6-edge.
Go module pinning
packages/system/kamaji/images/kamaji/Dockerfile
Adds a builder-stage RUN step that pins golang.org/x/net, golang.org/x/crypto, and go.opentelemetry.io/otel/sdk, then runs go mod tidy before compiling the manager binary.

Sequence Diagram(s)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested labels

security, area/build, size/M

Suggested reviewers

  • kvaps
  • lllamnyp
  • androndo
  • IvanHunters
  • sircthulhu
  • myasnikovdaniil

Poem

🐇 I nibbled the Dockerfile under the moon,
and pinned my modules to a steadier tune.
One tiny tag hop, one tidy little sigh,
now Kamaji builds as the stars go by.
Hoppity! 🌙

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: rebuilding Kamaji to address Go stdlib and module CVEs.
Linked Issues check ✅ Passed The PR bumps kamaji to 26.3.6-edge and patches Go dependencies, aligning with the release and CVE-fix goals.
Out of Scope Changes check ✅ Passed The changes stay within the requested image bump and security patching scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/kamaji-rebuild-stdlib-cve

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/XS This PR changes 0-9 lines, ignoring generated files area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug labels Jun 24, 2026
The kamaji source up to the 26.6.4-edge tag pins Go modules that carry
known advisories compiled into the manager binary: golang.org/x/net
below 0.55.0, golang.org/x/crypto below 0.52.0 and
go.opentelemetry.io/otel/sdk below 1.43.0. Unlike the distroless
OS-package findings, these advisories ship inside the Go binary and are
reported against the deployed image.

Force the fixed module versions during the build, after applying the
source patches and before compiling, so the binary embeds them. No edge
tag ships the fixes, and the tags that advance these modules also bring
the k8s 1.36 kubeadm change that regresses tenant control-plane RBAC, so
a source version bump is not a viable alternative.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei Aleksei Sviridkin (lexfrei) changed the title fix(kamaji): rebuild image to pick up Go stdlib security fixes fix(kamaji): close Go stdlib and module CVEs in the shipped binary Jun 24, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review June 24, 2026 18:35
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses multiple security vulnerabilities identified in the Kamaji manager binary. By updating the build source version and explicitly overriding specific Go module dependencies, the changes ensure that the resulting container image is free of known exploitable advisories that were previously present in the pinned dependencies.

Highlights

  • Build Version Update: Advanced the Kamaji build source from 26.3.5-edge to 26.3.6-edge to leverage the latest Go 1.26.4 toolchain and resolve standard library vulnerabilities.
  • Dependency Remediation: Forced updates for golang.org/x/net, golang.org/x/crypto, and go.opentelemetry.io/otel/sdk to patched versions during the build process to eliminate known CVEs in the compiled binary.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@dosubot dosubot Bot added area/build Issues or PRs related to image build infrastructure, multi-arch support security Security-related issues and features labels Jun 24, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Kamaji image version to 26.3.6-edge and overlays patched Go modules (golang.org/x/net, golang.org/x/crypto, and go.opentelemetry.io/otel/sdk) in the Dockerfile to address security vulnerabilities. The reviewer noted that the pull request description is missing the required release-note block as specified in the repository style guide, and reminded the author to ensure commits follow the Conventional Commits format and include a Signed-off-by: trailer.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

FROM golang:1.26 AS builder

ARG VERSION=26.3.5-edge
ARG VERSION=26.3.6-edge

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The pull request description is missing the required release-note block. According to the Repository Style Guide (lines 51-59), the PR body must contain a release note block in the following format:\n\ntext\nrelease-note\ntype(scope): human-readable changelog entry\n\n\n\nPlease update the PR description to include this block. Also, please ensure that all commits follow the Conventional Commits format and have a Signed-off-by: trailer.

References
  1. PR body must contain a release note block and commits must follow Conventional Commits format and have a Signed-off-by: trailer. (link)

@IvanHunters

Copy link
Copy Markdown
Collaborator

Verdict

LGTM with non-blocking notes.

Minimal, well-scoped image-rebuild patch: source bump 26.3.5-edge -> 26.3.6-edge plus a go get overlay placed correctly between git apply and go build, closing Go stdlib and three module advisories that the upstream tag still pins to vulnerable versions.

Findings

[NIT] packages/system/kamaji/images/kamaji/Dockerfile:1 — the golang:1.26 base floats, so the exact stdlib version embedded into the binary depends on whichever 1.26.x is current at image build time. Pre-existing across all sibling Dockerfiles in this repo, but worth tracking separately: pinning to golang:1.26.4 (or whichever patch is intended) would make the CVE-closing rebuild idempotent across release pipelines. Not a blocker.

Claim mismatches

  • [PARTIAL] PR body claims source bump alone cannot close module advisories because no edge tag up to 26.6.4-edge ships the fixed versions. Spot-checked 26.3.6-edge upstream go.mod: golang.org/x/net v0.51.0, golang.org/x/crypto v0.49.0, go.opentelemetry.io/otel/sdk v1.41.0 — matches PR body, overlay is necessary. Did not exhaustively walk every tag between 26.3.6 and 26.6.4, but the lower bound holds, which is sufficient for the argument.

Caveats

  • Binary / vendored chart drift. The Dockerfile builds from kamaji 26.3.6-edge, but packages/system/kamaji/charts/ is still vendored from 26.3.5-edge (last make update). Upstream diff 26.3.5-edge...26.3.6-edge adds an optional spec.networkProfile.advertiseAddress field to the TenantControlPlane CRD (feat: add advertiseAddress to NetworkProfile for split management/tenant addressing clastix/kamaji#1111) plus matching controller logic in internal/builders/controlplane/deployment.go and internal/resources/kubeadm_config.go. Verified existing-customer upgrade impact at code-path level: every new branch falls back to the prior address when advertiseAddress is unset, so for all in-cluster TenantControlPlane resources (which predate the field) the computed kube-apiserver Deployment spec, kubeadm endpoint, and cert SANs are byte-identical to the 26.3.5-edge binary. No rolling restart of tenant control planes, no cert regeneration. The only practical effect of the drift is that the new advertiseAddress feature is dormant in cozystack — set on a TenantControlPlane CR it gets silently pruned by the apiserver (standard CRD unknown-field pruning), so it appears configurable but is unreachable until make update refreshes the vendored chart. Worth either running make update to align the chart with the binary or noting the drift explicitly.
  • Strictly for the stated CVE goal, the source-tag bump is incidental — recompiling on the current golang:1.26 toolchain against the previous 26.3.5-edge source would close the stdlib advisories on its own. The overlay closes the module advisories independently of the source tag. Bumping the tag is reasonable for staying current with upstream, but the PR body presents it as required for the stdlib fix when it isn't.

Red-team concerns

  • Build determinism: kamaji upstream ships go.sum, go get @vX.Y.Z + go mod tidy mutates both go.mod and go.sum deterministically via MVS, so repeated builds against the same source tag produce the same module graph. No vendor/ directory means no parallel state to keep in sync. Acceptable.
  • Transitive bump scope: go mod tidy may pull additional indirect bumps beyond the three named modules; this is expected and is what makes the overlay valid.
  • Go version compatibility: target modules (x/[email protected], x/[email protected], otel/[email protected]) all declare go 1.25.0; upstream kamaji declares go 1.25.0; build toolchain is golang:1.26. No floor mismatch.
  • values.yaml still pins v1.5.0@sha256:130d… for both image.tag and extraArgs[0]=--migrate-image=…. Fine: the image target in packages/system/kamaji/Makefile re-writes both fields on release build with the new digest. No PR change to chart values needed.

Recommended follow-ups

  • Sibling Go-built images in packages/{core,system}/*/images/*/Dockerfile use the same floating golang:1.26[-alpine] base. If their first-party go.mod files import vulnerable x/net/x/crypto/otel/sdk versions, they need analogous bumps — but those live in the repo's own go.mod, so the correct lever is a direct dependency bump there, not a Dockerfile overlay. Out of scope for this PR; track separately if the scanner flags them.
  • Release-note fenced block: the PR template under .github/PULL_REQUEST_TEMPLATE.md expects a populated ```release-note ``` fenced block. The PR body has the heading but no fenced block. No CI enforcement, but worth filling in for changelog tooling.

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with non-blocking notes — full review in PR comment above.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 5be9bd5 into main Jun 25, 2026
17 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/kamaji-rebuild-stdlib-cve branch June 25, 2026 09:28
@lexfrei Aleksei Sviridkin (lexfrei) added area/kamaji Issues or PRs related to Kamaji (hosted control planes for tenant Kubernetes) and removed area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review labels Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Issues or PRs related to image build infrastructure, multi-arch support area/kamaji Issues or PRs related to Kamaji (hosted control planes for tenant Kubernetes) kind/bug Categorizes issue or PR as related to a bug security Security-related issues and features size/XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(kamaji): bump system/kamaji images to current upstream release

2 participants