fix(cert-manager): move the webhook off the kubelet's port - #3359

## What this PR does

The external-secrets webhook listens on port 10250 — the kubelet's port
— with `failurePolicy: Fail`. This is the same trap being removed from
cert-manager in #3359: a connection to the webhook Service that resolves
to a node IP instead of the pod IP reaches the kubelet, which completes
the handshake with its own node serving certificate, and the API server
rejects the call with an x509 error that blames the webhook's PKI rather
than the routing layer. The port collision does not cause the misroute;
it turns a rare transient one into a confidently misattributed failure.

The blast radius here is wider than admission alone:
`crds.conversion.enabled` defaults to true and points every
ExternalSecret CRD's conversion webhook at the same Service, so a
misrouted connection also breaks CRD conversion — reads at non-storage
versions, not just writes.

The webhook moves to 10260, matching #3359. The controller-runtime
default 9443 was considered and rejected: the etcd kube-rbac-proxy binds
it on host network, which would reproduce exactly the same
wrong-certificate failure with a different certificate. Nothing in the
tree binds 10260.

Unlike cert-manager, the upstream chart's Service targets the port
numerically — `targetPort: {{ .Values.webhook.port }}` — and a numeric
targetPort is copied into every endpoint verbatim rather than resolved
per pod. With one replica, a port change would direct traffic at a port
the still-running old pod does not serve while the replacement is not
yet Ready, failing every admission and conversion call in the gap;
surge-first rollout does not help because both pods sit behind the same
number. A one-line vendored-chart patch points the Service at the
`webhook` container-port name upstream already declares (the metrics
port in the same Service is already named), which closes the gap: during
the overlap each pod is routed on the port it actually serves. The patch
is applied by `make update` with `--forward`, so an already-applied hunk
fails loudly instead of being silently reversed. Upstream has already
made the same change — merged as external-secrets/external-secrets#5108
and shipped since chart 0.19.1 — but the vendored chart here is 0.10.4,
nine minor releases earlier, so the patch carries the fix until the
chart is bumped past that point. The `--forward` guard is exactly what
will flag that moment: on a bumped chart the patch fails loudly,
signalling it should be dropped.

external-secrets-operator ships disabled by default and is enabled
per-cluster via `bundles.enabledPackages`, so this is defensive
hardening for opt-in installs rather than a response to an observed
incident.

`metrics-server` is a third instance of the same port collision, left
out here: it serves an APIService rather than an admission webhook, a
lesser failure class deserving its own change.

### Release note

```release-note
fix(external-secrets): move the webhook off port 10250, which the kubelet also serves, and make its Service target the port by name so the move rolls out without an admission gap
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved webhook Service routing by switching to the named
`targetPort` (`webhook`) for pod traffic.
* Configured the webhook listener to run on port **10260** to avoid
prior connection/TLS certificate misrouting issues.
  * Preserved admission validation behavior and CRD conversion routing.
* **Tests**
* Added Helm chart regression tests covering webhook port wiring,
networking settings, failure policy behavior, and CRD conversion.
* **Chores**
* Added Helm chart unit testing to the development workflow and extended
the chart update process to apply the webhook routing adjustment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## What this PR does

metrics-server is the third component serving on port 10250 — the
kubelet's port — after the webhooks addressed in #3359 and #3360. It
ships default-enabled in the platform bundle and is also installed into
every tenant Kubernetes cluster with no value overrides, so a
package-root override propagates to both.

The hard defect here is different from the webhook cases. The chart has
first-class `hostNetwork.enabled` support, documented upstream for
clusters where the API server cannot reach pod IPs — and on the default
port that mode cannot start at all: the kubelet already holds 10250, so
the process dies with `bind: address already in use`. Upstream is aware
(kubernetes-sigs/metrics-server#1026 carries the exact panic) and
declined to change the chart default, advising hostNetwork users to set
their own port. This override is that port, set ahead of need.

The misleading-certificate angle of the sibling fixes mostly does not
apply: the chart defaults `apiService.insecureSkipTLSVerify` to true, so
a connection misrouted to a node would fail silently rather than with a
wrong-certificate error — and with endpoints resolving to pod IPs such a
misroute is not established to occur here at all. The comment states
that as a hypothesis, not a fact. What is factual is the consumer set:
`kubectl top`, tenant HPAs, and the console's cluster-usage page all
read `metrics.k8s.io`; the shipped VPA does not — it is configured with
Prometheus storage.

The port is 4443 rather than the siblings' 10260, for a reason worth
keeping: metrics-server is the one component of the three with supported
hostNetwork, and parking it on 10260 would put a live host-network
listener on exactly the port the webhook fixes rely on being silent —
turning their honest connection refusals back into answered handshakes.
4443 is the value metrics-server's own manifests shipped through v0.6.4,
before v0.7.0 moved them to 10250; it sits outside the 10248-10259
node-component block, and nothing in the tree binds it.

No chart patch is needed: the Service, both probes and the
ServiceMonitor already reference the port by name, so the rollout-gap
that required a vendored patch in #3360 does not exist here. The package
also gains the standard `test:` target it was missing.

Not verified without a cluster: that the aggregated APIService stays
Available through an upgrade, and that no supported substrate filters
4443 inbound on nodes — the tree configures no host firewall, but Talos
and managed substrates are outside what a render can prove. There is no
e2e coverage of `metrics.k8s.io` at all; a wrong guess here would
surface as HPAs silently not scaling.

### Release note

```release-note
fix(metrics-server): move the listener off port 10250, which the kubelet also holds. The default made hostNetwork mode unable to start at all, and the new port keeps node-port collisions out of the metrics path.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Configured the metrics server to listen on secure port 4443.
* Updated service routing and health checks to use the named HTTPS port,
improving reliability when port settings change.
  * Ensured pod networking does not rely on host networking.

* **Tests**
* Added automated chart tests covering container ports, service routing,
and liveness/readiness probes.
  * Added a test command for running the chart test suite.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…opes (#3362)

## What this PR does

PRs titled with the `cert-manager`, `external-secrets` or
`metrics-server` scopes landed on `area/uncategorized` because the
labeler's scope map had no entry for them — all three surfaced at once
in the current port-collision series (#3359, #3360, #3361).
metrics-server feeds the monitoring pipeline, so it maps to
`area/monitoring`; cert-manager and external-secrets are platform
services, so they map to `area/platform`.

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Chores**
  - Improved pull request labeling automation.
- Pull requests involving metrics, certificate management, and external
secrets are now categorized more accurately with the appropriate area
labels.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## What this PR does

The cert-manager webhook listens on port 10250 — the kubelet's port.
When a connection to the webhook Service resolves to a node IP rather
than the pod IP, it reaches the kubelet, which answers the TLS handshake
with its own node serving certificate. The API server then rejects every
cert-manager admission call in the cluster:

```text
failed calling webhook "webhook.cert-manager.io": ... x509: certificate is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc
```

The certificate is valid — it is simply the kubelet's, issued for the
node name. That signature is what separates this from the cainjector
failure mode, which reports `certificate signed by unknown authority`;
that possibility was checked and ruled out on the run where this was
observed. Here it blocked the whole platform bootstrap and pointed the
operator at cert-manager rather than at the routing layer.

The value comes straight from the vendored chart, where upstream chose
it deliberately for a different constraint: on GKE private clusters the
API server may only reach nodes on 443 and 10250. cozystack did not
override it.

Moving the webhook to 10260 is upstream's own remedy for this exact
error. The cert-manager troubleshooting guide documents the identical
signature — under a different cause, EKS Fargate, where each microVM
runs a kubelet that already holds 10250 — and prescribes `--set
webhook.securePort=10260`.

This does not fix the misroute, whose root cause is still unestablished;
the run carries no dataplane state, so candidates such as a stale
service map during bootstrap remain unproven. What it removes is the
trap that turns a rare, transient misroute into a cluster-wide failure
attributed to the wrong component. With the collision gone, the same
misroute surfaces as a connection refusal.

The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.

### Upgrade safety

The rendered webhook Deployment has one replica and no explicit
strategy, so the defaults apply: `maxUnavailable` 0 and `maxSurge` 1.
The replacement Pod must become Ready before the old one is removed. The
Service targets a named port rather than a number, so during the overlap
the EndpointSlice controller routes to each Pod on the port it actually
serves. There is no window with zero ready endpoints, and no maintenance
step is required. Had the Service pinned the port numerically, this
upgrade itself would have broken cluster-wide admission.

### Not addressed here

`external-secrets-operator` runs its webhook on 10250 with the same
`failurePolicy: Fail`, so it carries the identical exposure and deserves
the same treatment; `metrics-server` also uses 10250 but is an
APIService rather than an admission webhook. Both are left out to keep
this change to one component.

Closes #3355

### Release note

```release-note
fix(cert-manager): move the admission webhook off port 10250, which the kubelet also serves. A connection misrouted to a node answered with the kubelet's certificate, failing every cert-manager admission call cluster-wide with a misleading x509 error.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Configured the cert-manager webhook to use secure port 10260.
* Added coverage to verify secure port configuration, HTTPS service
routing, and disabled host networking.

* **Documentation**
* Added configuration comments explaining the secure port selection and
its role in avoiding service routing conflicts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

(cherry picked from commit 3affec9)
(cherry picked from commit ee8344d7bf782a608574ff3a064cb5c766ac49ff)

Backport of #3359 onto release-1.5.

The cert-manager webhook listened on 10250, the port the kubelet also
serves. When a connection to the webhook Service resolved to a node IP
rather than a Pod IP it reached the kubelet, which completed the TLS
handshake with its own node serving certificate, and the API server then
rejected every cert-manager admission call cluster-wide:

  failed calling webhook "webhook.cert-manager.io": ... x509: certificate
  is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc

The certificate is valid — it is simply the wrong server's — so the error
reads as a cert-manager fault and hides the misroute that caused it.
Moving the webhook to 10260 is upstream's own remedy for this signature.
It does not fix the misroute; it removes the collision that turns a rare
transient one into a cluster-wide outage attributed to the wrong
component.

The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.

Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY
packages/system/cert-manager/values.yaml, so every resource override in
that file arrived after v1.5.2. The bot's cherry-pick of this fix
conflicted for that reason alone — nothing to merge into — and applies
cleanly once the cainjector backport is in place.

Upgrade safety is unchanged from the original: one replica, no explicit
strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod
must be Ready before the old one goes. The Service targets a NAMED port,
so during the overlap each Pod is reached on the port it actually serves
and there is no window with zero ready endpoints.

Closes #3355

(cherry picked from commit 3affec9)

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

…belet's port (#3366)

# Description
Backport of #3359 to `release-1.5`.