CA cert in Secret not updated when self-signed CA itself gets renewed.

Describe the bug:
When using a self-signed issuer to managed an internal CA, when the CA itself gets renewed, none of the Certificates issued via the CA receive the updated CA cert, so once it expires all of the services using it fail to connect due to certificate expiration.

Expected behaviour:
When a CA managed by Cert-Manager is renewed, all Certificates issued by that CA should have their Secret updated with the new CA cert.

Steps to reproduce the bug:

Apply the following to a cluster

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
spec:
  isCA: true
  commonName: selfsigned-ca
  secretName: selfsigned-ca
  # Shortest time allowed
  duration: "1h"
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-ca
spec:
  ca:
    secretName: selfsigned-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-client
spec:
  secretName: selfsigned-client
  commonName: selfsigned-client
  issuerRef:
    name: selfsigned-ca
    kind: Issuer
    group: cert-manager.io

Note the CA fingerprint matches from the CA and client cert:

$ kubectl get secret selfsigned-ca -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -fingerprint -noout
SHA1 Fingerprint=7C:53:15:A9:EE:75:20:43:88:5A:5F:8C:AE:53:C0:D1:2A:77:77:A3

$ kubectl get secret selfsigned-client -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -fingerprint -noout
SHA1 Fingerprint=7C:53:15:A9:EE:75:20:43:88:5A:5F:8C:AE:53:C0:D1:2A:77:77:A3

Wait an hour for the CA certificate to be renewed...
Then check again:

$ kubectl get secret selfsigned-ca -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -fingerprint -noout
SHA1 Fingerprint=B2:48:49:D4:CC:45:F5:46:BF:B9:7D:AB:71:2C:2E:31:7E:7A:FD:59

$ kubectl get secret selfsigned-client -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -fingerprint -noout
SHA1 Fingerprint=7C:53:15:A9:EE:75:20:43:88:5A:5F:8C:AE:53:C0:D1:2A:77:77:A3

They no longer match, so the certificate bundle in the client Secret is wrong and workloads attempting to use it will see some sort of "certificate expiration" error.

Environment details::

Kubernetes version: v1.25.5
Cloud-provider/provisioner: N/A (bare-metal)
cert-manager version: v1.11.0
Install method: Helm via ArgoCD i.e. helm template ... | kubectl apply -f -

/kind bug

Thanks for raising this! As we discussed on slack this is a known issue, and one I'd like to address if at all possible.

I'm not sure that the expected behaviour you described would work, though: cert-manager cannot safely update already-issued certificates with the "new" CA after the CA is rotated - the certificates would have to be re-issued by the new CA.

Plus, we can't necessarily properly detect every certificate which would need re-issuing after the CA certificate is rotated.

I think the smallest useful change we could make would be to emit loud warnings when we're issuing a certificate which would expire after its issuer. That'll help people to detect when things are about to go wrong.

Beyond that, I think most of the solution to this lies in users planning their certificate hierarchy. It's possible for cert-manager to automate a lot of stuff in this space, but when a root certificate is rotated it can be dangerous for us to do anything at all. We could do a lot more towards documenting this!

cert-manager cannot safely update already-issued certificates with the "new" CA after the CA is rotated - the certificates would have to be re-issued by the new CA.

Not sure I follow this, but then of course I'm not familiar with the exact workflow of the controller(s).

When the CA Certificate is renewed, the backing Secret would also change I assume. The Issuer controller could/should be watching that Secret for changes and requeue all the Certificates that match its issuerRef for renewal also.

It's not so far from the current auto-reissue of Certificates and I'm not sure where the danger is in this process, but then I'm not a PKI expert so happy to be taught 😃

As a temporary workaround (in case others land on this issue), is just to increase the duration of the CA Certificate to something unreasonably long (8760h ~ 1 year), but I think something more robust is needed. At the very least an alterable metric for each Certificate when it's CA bundle is close to expiry.

Not sure I follow this, but then of course I'm not familiar with the exact workflow of the controller(s).

There's a whole lot of "distributed systems" stuff which can go wrong. We can look for all certs which refer to the changed issuer when the issuer is changed, but that process can fail leaving us in a halfway state where some certs won't be updated. Plus, there could be a lot of certificates, and reissuing them all could in the worst case involve generating a lot of large RSA keys which isn't cheap.

Even if we update all the certs successfully, they might not be picked up and used immediately by the pods which are mounting the secrets, adding more complexity.

Obviously I'm talking mainly about edge cases there and we it's reasonable to say that the happy path could still be improved even if all the edge cases aren't solved - but it's an illustration of the difficulties we have.

We also simply don't have that much time for making changes like this ATM. I'd love for cert-manager to be better at this stuff (it's super annoying to me that we don't have support for pathLen yet) but we just don't have that much time. 😦

just to increase the duration of the CA Certificate to something unreasonably long (8760h ~ 1 year)

I think this is likely to be a good approach - and scheduling in a calendar event to renew manually in say 9 months. It sounds very old-fashioned and not-cloud-native, but for non-trivial PKI stuff it often ends up with some manual steps (albeit usually because a PKI will have an offline root certificate meaning that rotation of intermediates is necessarily manual)

EDIT: To be clear, I'd rather automate it more than insist on manual steps, for sure. Just there are limits of how much we can safely automate (gotta consider that the CA certificate might also be trusted outside the cluster) and how much time we have to do this stuff

Thanks for the detailed response. I see a lot of the issues you're listing that to me would be more operational than functional i.e. "If you have a lot of certs managed by cert-manager, that's going to mean a lot of CPU overhead at renewal time" imho is a given.

We can look for all certs which refer to the changed issuer when the issuer is changed, but that process can fail leaving us in a halfway state where some certs won't be updated.

I see this as part of the normal reconciliation loop: every time the Certificate reconciliation is queued, check the issuerRef matches the ca.crt and if not, start a regular renewal with a CertificateRequest. Nothing should be lost half way for long, even with controller restarts.

Plus, there could be a lot of certificates, and reissuing them all could in the worst case involve generating a lot of large RSA keys which isn't cheap.

Understandable, but more an issue for operators. If lots of certs are currently managed by cert-manager the same more-or-less applies, and it's why we have rate-limited queues and CPU scheduling.

Even if we update all the certs successfully, they might not be picked up and used immediately by the pods which are mounting the secrets, adding more complexity.

Totally true and an existing issue solved by the current advice of using a tools such as "wave" if the pod process is incapable of detecting & reloading changed certs.

We also simply don't have that much time for making changes like this ATM.

Would you be open to a collaborative PR? As I mentioned in Slack, I work in Open Source too and luckily my company is very much open to upstream contributions. If the changes as I understand them are possible and could be accepted I could see it being a valuable investment of my time 😄

I don't mean to sound argumentative, just giving an outsiders perspective and making sure I fully understand the issues at hand 😃

@maelvls

I'm very open to collaboration and I'd gladly review a PR, but I'm also about to go on holiday for a while and realistically I won't personally be able to get back to doing anything until at least 2023-03-22 😦 After that, though, I'll be working a bunch on my Kubecon talk which is related to this use case so I should be able to help a bit then!

@maelvls is currently working on open source stuff and @irbekrm is too (although Irbe is currently on holiday), so they might be able to point you in the right direction if you have questions.

I'd suggest maybe starting with a design doc PR so we can agree on what changes we might want to make. Maybe also start a discussion in #cert-manager-dev on Kubernetes slack.

I hope that's helpful! I'll check back in when I'm back - and thank you for showing an interest, I love this part of open source 😁

Awesome, thanks for the info.

One implementation could be adding another check to the list of policies that the trigger controller checks before determining whether a cert needs to be re-issued. This check could just check that if the Certificate is for CA issuer (assuming then) check the issuer's CA secret and verify the the issued certificate matches the CA. We should then also update this event handler predicate as at the moment the trigger handler only runs on Certificate Secret events- the event handler predicate would be crucial to ensure that both the controller does get triggered when the CA Secret changes and does not get triggered (and run all the policy checks, some of which are expensive) when any random cluster Secret changes.

The above approach has a problem in that it would introduce more coupling between certificates controllers and the CA issuer, which may make any future refactoring difficult and does not fit the model well.

Alternative approach could be to add a new status field (i.e 'revision') that any issuer could implement that would signal to trigger controller that certificates need to be re-issued together with a new annotation for Certificates to signal for which 'revision' the Certificate was last successfully issued. With this, CA issuer could store something like CA cert fingerprint on its status and when it detects that a new CA cert has been stored in the Secret, it could bump the revision. We would add a new check to the trigger controller to trigger new issuance if the revision annotation does not match the revision field on the Certificate. That would allow us to avoid adding any issuer-specific logic to certificates controllers and any issuer, including external issuers, could implement their own logic for what would be considered a new 'revision' for them.
I've not thought about this in detail, but perhaps this approach would work. The question of whether something like this can be safely switched on in GA for all CA issuers still stands (i.e whether everyone wants this feature). Perhaps the CA issuer could have some new fields to allow users to specify if the revision needs bumping or not (perhaps too hacky). In theory, the issuer could have all kinds of logic, i.e to allow users to specify how long before expiry of CA cert all certs need to be renewed, to add some skew to avoid thundering herd of renewals etc, but that would be lots of extra complexity.
I'd be interested to see someone explore this approach in a design doc.

The third alternative would be to have some external component that applies Issuing condition when it detects that the CA certificate has been updated. That would need to either somehow store the fingerprint of the old certificate or have other means to distinguish between Secret updates because the CA certificate changed from other updates (i.e user added a new label).
The benefit of this is that the approach could be tested out and refined before adding the code in-tree.

Is it OK that CA issues certificates expiring after CA expiration? Does fixing #5864 will partially help to avoid this issue?

Change certificates expiration from 14 to 13 days

Change certificates expiration from 14 to 13 days (#758)

I feel like the trust manager is the solution to this problem - updating just the ca.crt in previously issued client secrets would mean the chain is broken in that secret. Reissuing all the certs could take a long time and cause a thundering herd style scenario. You really need a way to distribute both the old CA cert and the new CA cert until such time as all certs have been refreshed with the new CA.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

/remove-lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
/remove-lifecycle stale

@Jamstah

I feel like the trust manager is the solution to this problem - updating just the ca.crt in previously issued client secrets would mean the chain is broken in that secret. Reissuing all the certs could take a long time and cause a thundering herd style scenario. You really need a way to distribute both the old CA cert and the new CA cert until such time as all certs have been refreshed with the new CA.

Just gave this a try and it seems like trust-manager does help with this.

Here's the setup I tried

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca-short
  namespace: my-trust-namespace
spec:
  isCA: true
  commonName: selfsigned-ca-short
  secretName: ca-short
  duration: 1h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer-short
spec:
  ca:
    secretName: ca-short
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: bundle
spec:
  sources:
  - secret:
      name: ca-short
      key: tls.crt
  target:
    configMap:
      key: ca.crt

I first wanted to test the "month 8 case" as mentioned in this comment from a related issue: #2478 (comment)

For my test case since the CA expires each hour, the valid window is from the 40th to 60th minute. I also had a MQTT broker service running on an issued leaf cert from this CA with expiry of 30 days.

Around the 40th minute, my observations:

cert-manager successfully renews the CA and trust-manager picks it up and distributes the new trust bundle correctly.
The leaf certificate isn't renewed, with Kubernetes reporting that its last transition time was when it was issued for the first time. Given the OP of the issue, this is expected.

I create an MQTT client pod with mosquitto client and mounted the renewed trust bundle as a file then passed it in via --ca-file. I try sending a message to the MQTT broker, and it works!

$ mosquitto_pub -h my-broker -p 8884 -m "hello" -t "source" -d --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/mq-sat)
Client null sending CONNECT
Client $server-generated/393a34cb-f8a3-4b69-9a9f-4bc666176da9 received CONNACK (0)
Client $server-generated/393a34cb-f8a3-4b69-9a9f-4bc666176da9 sending PUBLISH (d0, q0, r0, m1, 'source', ... (5 bytes))
Client $server-generated/393a34cb-f8a3-4b69-9a9f-4bc666176da9 sending DISCONNECT

I use openssl and can verify the leaf cert chain validity with both the old CA and new CA

$ openssl verify -verbose -CAfile new-ca.crt broker-leaf.crt
broker-leaf.crt: OK
$ openssl verify -verbose -CAfile old-ca.crt broker-leaf.crt
broker-leaf.crt: OK

I think this works because, by default, cert-manager doesn't rotate the private key when renewing a cert, CA or not. Combine it with what @Jamstah said in this comment #5864 (comment)

...leaf certs are signed by the CA key and not by the CA cert. The same CA key can be used to create a new CA cert with new validity dates that would be able to be used in the chain with the leaf cert and validate it successfully, even though the original CA cert has expired. There's a good description here: https://serverfault.com/questions/878919/what-happens-to-code-sign-certificates-when-when-root-ca-expires

So in my case since the private key stayed the same, the leaf cert issued from the old CA can be cryptographically chained to the new CA even though the public key is different. I think this also means that as long as the client gets the newest trust bundle, the leaf cert can be used even though it way outlives the original CA (30 days vs 1 hour).

Lastly, after 60 minutes, the original CA expired, but I can still send messages and pass openssl verification as long as I use the renewed trust bundle. If I use the old trust bundle, I get an error from mosquitto saying OpenSSL Error[0]: error:0A000086:SSL routines::certificate verify failed and openssl verify fails with error 10 at 1 depth lookup: certificate has expired / error broker-leaf.crt: verification failed. Shows how important trust manager is.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close

@cert-manager-bot

@cert-manager-bot: Closing this issue.

Details

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

/reopen

@ebuildy

@ebuildy: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

I encountered the same issue and resolved it by extending the duration of the self-signed CA, ensuring that the renewBefore period is longer than the duration of the issued certificate. This guarantees that the signed certificate always has a valid (non-expired) CA certificate.

For example:

Self-signed CA: duration of 6 months, with renewBefore set to 3 months
Issued Certificate: duration of 3 months, with renewBefore set to 1 day

Increase duration of CA certificate

DENA-1405 keda down due to certificate expiry: increase the validity …

@NoodlesWang2024

e: duration of 3 months, with renewBefore set to 1 day

I encountered the same issue and resolved it by extending the duration of the self-signed CA, ensuring that the renewBefore period is longer than the duration of the issued certificate. This guarantees that the signed certificate always has a valid (non-expired) CA certificate.

For example:

Self-signed CA: duration of 6 months, with renewBefore set to 3 months

Issued Certificate: duration of 3 months, with renewBefore set to 1 day

@NoodlesWang2024 can you please explain how can child cert renewing before CA cert renewal will help resolve problem here?
Considering the example given, starting both cert duration from 0th day, on 2months 29th day (1 day before 3 months), child certificate will be renewed with same CA cet as earlier on 0th day and on completing 3 months i.e. next day, CA cert will be renewed. So now any service using the renewed child certificate will fail the TLS communication, since cert verification will fail as CA cert used to sign child cert is different than current CA cert (renewed one).
Please correct if anything missed here.

@akankshapanse

@akankshapanse [on day 89] child certificate will be renewed with same CA cet as earlier on 0th day and on completing 3 months i.e. next day, CA cert will be renewed. So now any service using the renewed child certificate will fail the TLS communication, since cert verification will fail as CA cert used to sign child cert is different than current CA cert (renewed one).

Call first CA CA1. On day 0 we generate leaf cert L1. Cert-manager saves both to a Secret resource as ca.crt = CA1 and tls.crt = L1, alongside tls.key for the private key.

On day 89, we generate L2 and sign it with CA1. Both are once again saved to the Secret.

On day 90 we generate CA2. The k8s secret is untouched. It still contains ca.crt = CA1 and tls.crt = L2. CA1 is valid for another 90 days and L2 is valid for another 89 days. Absolutely nothing references or starts validating against CA2 yet; both servers and clients consult the ca.crt in the Secret, which has not changed.

On day 178, we renew leaf again. It picks up CA2 at last.

Etc. The thing you missed is that ca.crt is stored alongside tls.crt and the two files in the secret are always valid with respect to each other. I do not recommend having those particular durations because it's cutting it really fine, you cannot tolerate any delay like cert-manager going down for a few hours. Make CA renewBefore >= 2x leaf duration or similar safety margin. That ensures leaf renews twice in the time between CA1 being superseded (by CA2) and CA1 expiring.

There are other aspects of cert validity that are not covered by just setting durations. You need to ensure applications are not using stale certs or private keys. Many applications will read those files over and over again assuming they'll be rotated, which is good. Your average database client will do this, that's why you usually configure the "cafile" as a file path. But some software just has to be restarted.

Finally, a client validating with a given ca.crt will often see a server switch both the leaf and CA at the same time (eg at day 178 above). This is bad because you cannot ensure the server and client get updated secret volume mounts and begin using them at the same instant. Without some way of serving both CA1 and CA2 in the same ca.crt, you will have a short period (not 3 months, closer to minutes) while clients or servers are lagging or restarting etc. cert-manager's ca.crt files only ever contain the one cert used to sign the given leaf. This is the problem that trust-manager solves by maintaining a cert bundle of all currently valid CA certs so that clients pick up a bundle containing CA1 + CA2 long before L2 is actually issued. It is a shame that cert-manager can't do it.

Navigation Menu

Repository navigation

CA cert in Secret not updated when self-signed CA itself gets renewed. #5851

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CA cert in Secret not updated when self-signed CA itself gets renewed. #5851

Description

Activity

SgtCoDFish commented on Mar 6, 2023

TimJones commented on Mar 7, 2023

SgtCoDFish commented on Mar 7, 2023

TimJones commented on Mar 7, 2023

SgtCoDFish commented on Mar 7, 2023

TimJones commented on Mar 7, 2023

irbekrm commented on Mar 13, 2023

n9 commented on Apr 26, 2023

Jamstah commented on May 18, 2023

jetstack-bot commented on Aug 16, 2023

n9 commented on Aug 17, 2023

22 remaining items

cert-manager-bot commented on Jul 4, 2024

jlian commented on Jul 26, 2024

cert-manager-bot commented on Aug 25, 2024

cert-manager-prow commented on Aug 25, 2024

ebuildy commented on Sep 24, 2024

cert-manager-prow commented on Sep 24, 2024

NoodlesWang2024 commented on Oct 15, 2024

akankshapanse commented on Oct 8, 2025

cormacrelf commented on May 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions