Repository navigation
Webhook inject-ca-from annotation causes downtime #6350
Description
We have the following setup:
A certificate for mTLS between the API server and a validating webhook:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: serving-cert
spec:
dnsNames:
- my-service.{{ .Release.Namespace }}.svc
- my-service.{{ .Release.Namespace }}.svc.cluster.local
issuerRef:
kind: Issuer
name: selfsigned-issuer
secretName: webhook-server-cert
duration: 1h0m0s
renewBefore: 59m0sThe validating webhook:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: "my-service"
annotations:
cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/serving-cert
webhooks:
- ...We have started investigating a problem after the webhook would no longer be called successfully. The service implementing the validating webhook repeatedly throws the following error in the log:
http: TLS handshake error from <IP>: remote error: tls: bad certificate
The error occurs whenever the certificate is renewed. When the certificate is renewed by cert-manager both the secret and the caBundle field in the ValidatingWebhookConfiguration are updated by cert-manager. It is up to the application to serve the new certificate from the updated secret.
One solution to this is the certwatcher package.
However, this alone does not fully resolve one specific problem. Whenever the certificate is renewed the changes to the ValidatingWebhookConfiguration's caBundle and the secret don't occur at exactly the same time.
- the ValidatingWebhookConfiguration's caBundle seems to change nearly instantly
- it takes up to a minute between the certificate renewal and the updated secret content showing up in the mounted secret in the application.
During this time, the application still serves the old certificate while the API server ONLY accepts the new certificate.
Expected behaviour:
The caBundle should contain both old and new certificate so there is no race condition between the secret content updating on disk and the caBundle being updated in the ValidatingWebhookConfiguration. Shortly before the old certificate's expiry the old certificate it should be removed from the caBundle.
Steps to reproduce the bug:
With the manifests above the certificate is renewed once per minute.
You can verify the caBundle injected into the ValidatingWebhookConfiguration only ever contains one certificate. This can only work if the API server client updates the certificate they serve in the same instant as the caBundle changes.
Anything else we need to know?:
It seems there have been previous discussions on this error but noone pointed out this problem specifically.
Environment details::
- OpenShift 4.12 / Kubernetes 1.25
- cert-manager version: 1.13.0
- Install method: helm
/kind bug
Activity
SgtCoDFish commentedon Sep 20, 2023
Hey, thanks for raising this!
We've definitely had some discussions around improving how we surface trusted CAs - as you correctly point out, the issue here is that the caBundle in this scenario would need to trust both the old and new certs for a time in order to rotate. In some situations trust-manager helps with this, but not all.
I think that discussion is valuable, but it's a larger discussion. In this case, my approach would be slightly different: I'd either issue a very long-lived self-signed cert to remove the problem, or I'd issue a long-lived self-signed cert which in turn issues shorter certificates.
Essentially I'd try to follow this guide.
Using the examples in those docs, the webhook would have my-selfsigned-ca injected, and the client would use a cert issued from my-ca-issuer.
I think that setup would be acceptable to me - would it work for you?
BWagenerGenerali commentedon Sep 26, 2023
Thank you, that's a great workaround!
We too opted for the long-lived self-signed cert for now. But I prefer the solution you laid out and linked to.
For us this works well enough, now that we understand the problem.
As the problem is hard to track down and understand, it would be nice if it could be avoided. I'm not sure if this edge case is worth pointing out in documentation. Maybe until the behavior of the caBundle injection has been changed (if this is something you are considering).
Cheers!
jetstack-bot commentedon Dec 25, 2023
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to jetstack.
/lifecycle stale
jetstack-bot commentedon Jan 24, 2024
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with
/remove-lifecycle rotten.Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with
/close.Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale
jetstack-bot commentedon Feb 23, 2024
Rotten issues close after 30d of inactivity.
Reopen the issue with
/reopen.Mark the issue as fresh with
/remove-lifecycle rotten./close
jetstack-bot commentedon Feb 23, 2024
@jetstack-bot: Closing this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.