The cert-manager webhook listens on port 10250 — the kubelet's port. When a request to the webhook Service resolves to a node IP instead of the pod IP, it reaches the kubelet, which answers with its node serving certificate, and the API server rejects every cert-manager admission call cluster-wide with a thoroughly misleading error.
Observed on an E2E bootstrap run:
failed calling webhook "webhook.cert-manager.io": ... x509: certificate is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc
The certificate is valid — it is just the kubelet's, issued for the node name. That signature is what distinguishes this from the known cainjector-OOM / uninjected-caBundle failure, whose error is certificate signed by unknown authority; that hypothesis was checked first and ruled out on this run (no OOM, no cainjector restarts, the 512Mi limit already in place).
The mechanism: the vendored chart ships webhook.securePort: 10250, the webhook Service targets it, and the upstream values.yaml itself warns two lines above that this collides with the kubelet in host-network setups. Cozystack does not override it — packages/system/cert-manager/values.yaml only sets resources. So any misroute of the Service's traffic to a node IP does not fail loudly; it lands on a live TLS server that answers with the wrong certificate, and the resulting x509 error points the operator at cert-manager instead of at the routing layer. On this run it blocked the entire platform bootstrap.
What has not been established is why the connection resolved to the node IP in the first place — the log carries no EndpointSlice or dataplane state, so candidates like a stale service map during bootstrap remain unproven. The port collision is not the root cause of the misroute; it is what turned a transient misroute into a cluster-wide, wrongly-attributed failure.
Proposed hardening, independent of the root cause:
cert-manager:
webhook:
securePort: 10260With the collision gone, the same misroute becomes a connection refusal — an honest, immediately-diagnosable error instead of a valid-but-wrong certificate. The root cause of the misroute deserves its own investigation with dataplane state captured; this change just removes the trap that makes it catastrophic and misleading.
Frequency note: across seven E2E runs from the same two days, the signature appears in exactly one — so the misroute itself is rare, which makes the misleading failure mode all the more expensive when it fires.
The cert-manager webhook listens on port 10250 — the kubelet's port. When a request to the webhook Service resolves to a node IP instead of the pod IP, it reaches the kubelet, which answers with its node serving certificate, and the API server rejects every cert-manager admission call cluster-wide with a thoroughly misleading error.
Observed on an E2E bootstrap run:
The certificate is valid — it is just the kubelet's, issued for the node name. That signature is what distinguishes this from the known cainjector-OOM / uninjected-caBundle failure, whose error is
certificate signed by unknown authority; that hypothesis was checked first and ruled out on this run (no OOM, no cainjector restarts, the 512Mi limit already in place).The mechanism: the vendored chart ships
webhook.securePort: 10250, the webhook Service targets it, and the upstreamvalues.yamlitself warns two lines above that this collides with the kubelet in host-network setups. Cozystack does not override it —packages/system/cert-manager/values.yamlonly sets resources. So any misroute of the Service's traffic to a node IP does not fail loudly; it lands on a live TLS server that answers with the wrong certificate, and the resulting x509 error points the operator at cert-manager instead of at the routing layer. On this run it blocked the entire platform bootstrap.What has not been established is why the connection resolved to the node IP in the first place — the log carries no EndpointSlice or dataplane state, so candidates like a stale service map during bootstrap remain unproven. The port collision is not the root cause of the misroute; it is what turned a transient misroute into a cluster-wide, wrongly-attributed failure.
Proposed hardening, independent of the root cause:
With the collision gone, the same misroute becomes a connection refusal — an honest, immediately-diagnosable error instead of a valid-but-wrong certificate. The root cause of the misroute deserves its own investigation with dataplane state captured; this change just removes the trap that makes it catastrophic and misleading.
Frequency note: across seven E2E runs from the same two days, the signature appears in exactly one — so the misroute itself is rare, which makes the misleading failure mode all the more expensive when it fires.