Skip to content

[Backport release-1.5] fix(cert-manager): move the webhook off the kubelet's port - #3366

Merged
myasnikovdaniil merged 2 commits into
release-1.5from
backport-3359-to-release-1.5
Aug 4, 2026
Merged

myasnikovdaniil merged 2 commits into
release-1.5from
backport-3359-to-release-1.5

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3359 to release-1.5.

@github-actions

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3359-to-release-1.5
git worktree add --checkout .worktree/backport-3359-to-release-1.5 backport-3359-to-release-1.5
cd .worktree/backport-3359-to-release-1.5
git reset --hard HEAD^
git cherry-pick -x df85632c8909e775580a7c8732dffae8b8e473ca
git push --force-with-lease

…injection

cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.

Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.

Signed-off-by: Ivan Okhotnikov <[email protected]>
(cherry picked from commit c3afc63)
(cherry picked from commit 9f312d7a216c34724848142415d2de7c086a711e)
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-3359-to-release-1.5 branch from 1e7474e to ea15a20 Compare July 31, 2026 05:28
@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug labels Jul 31, 2026
Backport of #3359 onto release-1.5.

The cert-manager webhook listened on 10250, the port the kubelet also
serves. When a connection to the webhook Service resolved to a node IP
rather than a Pod IP it reached the kubelet, which completed the TLS
handshake with its own node serving certificate, and the API server then
rejected every cert-manager admission call cluster-wide:

  failed calling webhook "webhook.cert-manager.io": ... x509: certificate
  is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc

The certificate is valid — it is simply the wrong server's — so the error
reads as a cert-manager fault and hides the misroute that caused it.
Moving the webhook to 10260 is upstream's own remedy for this signature.
It does not fix the misroute; it removes the collision that turns a rare
transient one into a cluster-wide outage attributed to the wrong
component.

The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.

Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY
packages/system/cert-manager/values.yaml, so every resource override in
that file arrived after v1.5.2. The bot's cherry-pick of this fix
conflicted for that reason alone — nothing to merge into — and applies
cleanly once the cainjector backport is in place.

Upgrade safety is unchanged from the original: one replica, no explicit
strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod
must be Ready before the old one goes. The Service targets a NAMED port,
so during the overlap each Pod is reached on the port it actually serves
and there is no window with zero ready endpoints.

Closes #3355

(cherry picked from commit 3affec9)

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@dosubot dosubot Bot added the kind/backport Categorizes issue or PR as requiring a backport to the current release line label Aug 4, 2026
@myasnikovdaniil
myasnikovdaniil merged commit 95bb827 into release-1.5 Aug 4, 2026
10 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3359-to-release-1.5 branch August 4, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/backport Categorizes issue or PR as requiring a backport to the current release line kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants