Describe the bug
Certificates the opensearch operator issues itself last one year and are never reissued, so a release that stays on the operator issuer stops serving a valid certificate a year after it first converged, with nothing on the objects to say so.
At the pinned v2.8.0, pkg/tls/pki.go:152 gives every leaf NotAfter: time.Now().AddDate(1, 0, 0) while the CA at pki.go:62 gets ten years. All three generate paths in pkg/reconcilers/tls.go create a Secret only when reading it failed, that is only when it does not exist: handleHttp at 490 to 492, handleTransportGenerateGlobal at 272 to 274, and handleTransportGeneratePerNode at 329 to 331. None of them looks at the expiry of what is already there.
The one certificate the operator does renew by age is the admin certificate: shouldCreateAdminCert builds tls.NewCertValidater(data, tls.WithExpiryThreshold(5*24*time.Hour)) and branches on IsExpiringSoon() (tls.go:191 and :196).
So a year in, the HTTP listener on 9200 presents an expired certificate, securityadmin stops authenticating against it, and transport mTLS between nodes is on the same clock and cannot be turned off. No event is recorded and no condition changes.
Environment
- Cozystack main, opensearch-k8s-operator pinned at v2.8.0
- Any OpenSearch release whose certificates come from the operator
To Reproduce
The failure needs a release older than a year, which no test environment here has reached, so this is read from the source rather than reproduced on a cluster. The shape is visible without waiting: the generate paths above are guarded on the Secret's absence alone.
Expected behaviour
Either the operator reissues a leaf before it expires, the way it already does for the admin certificate, or the leaves are issued with a lifetime that matches the CA.
Actual behaviour
The leaf is minted once and kept until something deletes the Secret.
Additional context
Found while reviewing #2682. That branch narrows the blast radius rather than causing it: for externally published releases it moves the HTTP and Dashboards certificates to a per-release cert-manager CA with a lifetime aligned to that CA, so what is left on the operator path afterwards is the cluster-internal releases and transport mTLS everywhere.
Workaround until it is fixed upstream: delete the certificate Secret and let the operator mint a new one, which also rolls the pods that mount it.
Checklist
Describe the bug
Certificates the opensearch operator issues itself last one year and are never reissued, so a release that stays on the operator issuer stops serving a valid certificate a year after it first converged, with nothing on the objects to say so.
At the pinned v2.8.0,
pkg/tls/pki.go:152gives every leafNotAfter: time.Now().AddDate(1, 0, 0)while the CA atpki.go:62gets ten years. All three generate paths inpkg/reconcilers/tls.gocreate a Secret only when reading it failed, that is only when it does not exist:handleHttpat 490 to 492,handleTransportGenerateGlobalat 272 to 274, andhandleTransportGeneratePerNodeat 329 to 331. None of them looks at the expiry of what is already there.The one certificate the operator does renew by age is the admin certificate:
shouldCreateAdminCertbuildstls.NewCertValidater(data, tls.WithExpiryThreshold(5*24*time.Hour))and branches onIsExpiringSoon()(tls.go:191and:196).So a year in, the HTTP listener on 9200 presents an expired certificate, securityadmin stops authenticating against it, and transport mTLS between nodes is on the same clock and cannot be turned off. No event is recorded and no condition changes.
Environment
To Reproduce
The failure needs a release older than a year, which no test environment here has reached, so this is read from the source rather than reproduced on a cluster. The shape is visible without waiting: the generate paths above are guarded on the Secret's absence alone.
Expected behaviour
Either the operator reissues a leaf before it expires, the way it already does for the admin certificate, or the leaves are issued with a lifetime that matches the CA.
Actual behaviour
The leaf is minted once and kept until something deletes the Secret.
Additional context
Found while reviewing #2682. That branch narrows the blast radius rather than causing it: for externally published releases it moves the HTTP and Dashboards certificates to a per-release cert-manager CA with a lifetime aligned to that CA, so what is left on the operator path afterwards is the cluster-internal releases and transport mTLS everywhere.
Workaround until it is fixed upstream: delete the certificate Secret and let the operator mint a new one, which also rolls the pods that mount it.
Checklist