Repository navigation
e2e: opensearch admin certificate is intermittently not re-minted under the chart CA within the step's budget #4332
Description
Activity
The
cozyreportartifact of a failing run does answer this, which corrects what I wrote above: the operator's log is collected, just not by thecatchsteps. The crust-gather host snapshot taken in the same catch carries it, atsnapshots/opensearch/host/namespaces/cozy-opensearch-operator/.... It is simply not visible in the job log, so it takes a 16MB download to read.What it shows changes the diagnosis. The re-mint is not late. The reconcile loop stops.
Run 35343282947 (the #4326 one):
- the operator reconciles
OpenSearchClusteron a steady cadence, gaps of 10.2s
for almost every one of its 27 reconciles, the widest gap in the whole run 26.5s - its last log line is at 13:51:33.225Z
- helm-controller applies the new CR revision, generation 3, the one that moves
the cluster onto the chart CA, at 13:51:35Z - after that, nothing at all for the remaining 240s of the step. No
Reconciling OpenSearchCluster, and noadmin cert is not signed by CA, recreating - the pod is Running and Ready throughout, with 0 restarts
Run 35333289958, a different branch and a different operator pod, has the same shape: last line 10:56:35.706Z, helm-controller Apply at 10:56:38Z, widest gap before that 24.2s, then silence.
Two things I checked and ruled out in the failing run: the securityconfig Job completed successfully at 13:51:32, and the cluster itself is quiet in the window, with no TLS errors attributable to the operator.
So the step's 240s budget is not what fails here. Nothing is going to move that certificate, because the controller stops reconciling the object at the moment the chart rewrites it, and does not come back.
Where it stops is the open question. The chain in v2.8.0 is
tls,securityconfig,config,cluster,scaler,dashboards,upgrade,restart,snapshotrepository, and only the TLS one logs on entry. The last line in both runs is itsGenerating certificates, so everything after that is unobservable. There is no goroutine dump in the artifact either.Next step I plan: a patch on the image that logs each component reconciler as it is entered and left, so the next occurrence names the one that hangs instead of leaving a 240s hole.
- the operator reconciles
The operator stopped reconciling, and the log in run 35333289958 shows it.
Reconciling OpenSearchClusterevery ~10s from 10:31:31, last line at 10:56:35.706, then nothing until collection at 11:01:11. The log is complete: 144 lines under--tail 2000, starting atStarting manager, no container restart. So the silence is real, 4m36s with no reconcile, covering the step's window of 10:56:39 to 11:00:43. The TLS reconciler runs first incomponentReconcilers, so it runs on every pass. No pass ran, and the 240s budget is not what failed.The same log shows where the pass went. The last thing before the silence is the cluster reconciler,
Excluded opensearch-test-bootstrap-0 from voting and removed the bootstrap podat about 10:56:36. Worker count is 1, so one blocked call stops everything behind it. The candidates are the reconcilers that talk to the OpenSearch HTTP API whilenodes-0was NotReady.The discriminator is already in the runs, and it cuts both ways. In both red runs the cleanup afterwards hangs for exactly five minutes and fails with
OpenSearchCluster/tenant-test/opensearch-test termination timeout: context deadline exceeded(13:56:09 to 14:01:09, and 11:01:12 to 11:06:12), through the same finalizer and the same worker. In the green run cleanup takes seconds. One stuck worker explains both the certificate that was never re-minted and the delete that never finished.That also makes this the same defect as the earlier observation on the roll, where the StatefulSet still carried the pre-move
http-certvolume while the CR already named both cert-manager Secrets and both Certificates were Ready. The re-mint and the mount-plan rewrite come out of the same pass, so those are two views of one missing reconcile rather than two bugs.Four corrections, none of which change the conclusion:
verify-statefulset-rolled-onto-the-new-mount-plandid not pass in the red runs. It sits after the admin-cert step and never ran, appearing zero times in either job log. The separation from e2e: opensearch times out waiting for a StatefulSet roll that never starts #4259 still holds, on the failing step rather than on that.- The catch did collect the operator log in run 35333289958. Its head carries a second
podLogsleg oncozy-opensearch-operatorwithcontrol-plane=controller-manager. That leg is absent on main and in run 35343282947, so the gap is real today, just not in the run that turned out to hold the answer. Health: greenis from 35343282947. In 35333289958 it readsHealth: unknown, and both are stale, since the suite had made the cluster yellow a minute earlier.- 759s belongs to a different row. The opensearch case in 35343282947 is 459.08s, and 755.48s is the whole report of 35333289958.
I could not reproduce the two-of-eight count. In-tree jobs that day that actually ran the opensearch suite on a post-#4317 tree come to eleven, ten at the time of filing. The numerator holds.
One more for the record, found on the way:
IsSignedByCAdoes not verify a signature. It compares DER bytes,bytes.Equal(cert.RawIssuer, caCert.RawSubject).A lead on where the reconcile parks, found by Aleksei Sviridkin (@lexfrei) while reviewing #4333 and checked against the v2.8.0 source here.
The operator's OpenSearch client has no client-side timeout.
NewOsClusterClientbuilds its transport as a barehttp.TransportwithTLSClientConfig,DisableKeepAlivesandMaxIdleConnsand noTimeouton thehttp.Clientaround it (opensearch-gateway/services/os_client.go:72-86), and the calls go out oncontext.Background()rather than the reconcile's context.ClusterHealthRequest.Timeoutat os_client.go:281 is not a counterexample: opensearch-go v1.1.0 renders it into the request's query string, so it bounds how long the server waits, not how long the operator does.CheckClusterStatusForRestartis one of those calls, reached fromrestart.ReconcilethroughrestartStatefulSetPod(pkg/reconcilers/rollingRestart.go:177).SetupWithManagersets noMaxConcurrentReconciles, so the default of one applies: a single call that never gets an answer parks the only worker, and every later event for the object queues behind it. That matches the shape here exactly, including the part that puzzled me, namely a pod that stays Running and Ready while reconciling nothing.Not proven for these two runs. What would prove it is the component name, which is what #4333 is for: with the watchdog in, a stall writes
component reconciler has not returnedwith the component and the elapsed, once a minute, for as long as it lasts.- added a commit that references this issue
on Sep 23, 2026 - added a commit that references this issue
on Sep 24, 2026
Describe the bug
The
opensearchchainsaw suite fails intermittently atverify-admin-certificate-moved-to-the-chart-ca. After the CR is moved onto the cert-manager issued CA, the step pollsopensearch-test-admin-certfor 240s waiting for its issuer to become the chart CA, and on some runs that never happens. On other runs, including the same branch two hours later, the same step completes in about 30 seconds.This is not #4259. That one is the
verify-statefulset-rolled-onto-the-new-mount-planstep, its cause was the operator's restart gate on a permanently yellow cluster, and #4317 fixed it. Every occurrence below is on a tree that already contains #4317, and in all of them the mount-plan step passed.Environment
main(with fix(opensearch-operator): let a permanently yellow cluster restart #4317)To Reproduce
Run the full chainsaw suite. The
opensearchsuite fails atverify-admin-certificate-moved-to-the-chart-caafter about 250s, with the whole suite taking roughly 760s instead of the 360s a green run takes.Expected behaviour
Once the CR names the chart's CA, the operator re-mints the admin certificate under it, and the step sees the new issuer well inside its budget, as it does on green runs.
Actual behaviour
The issuer stays on the operator's own CA until the deadline:
Nothing else in the failing run looks unhealthy. The HelmRelease is Ready, the cluster reports
Health: green,Phase: RUNNING, both Certificates are issued, and the nodes StatefulSet has finished rolling. The admin-cert step is the only ERROR in the test.Occurrences on 2026-09-18, all on branches confirmed to contain #4317:
Two of the eight in-tree jobs that day with a decided outcome and a confirmed post-#4317 tree failed this way. The last two rows are the same branch, so the failure is not a property of what is being tested: #4326 only adds a
go teststep to builder stages and does not change any operator binary.Additional context
The re-mint decision lives upstream in
opensearch-operator/pkg/reconcilers/tls.goat v2.8.0.shouldCreateAdminCertcompares the stored certificate against the CA returned bygetCACert, and when the CR names an HTTP CA secret, that is the chart's CA, so the comparison fails and the operator logsadmin cert is not signed by CA, recreating. None of the three patches Cozystack carries for this image touches that path.Corrected after the fact, and the comments below carry the detail: the failing runs do answer this. The operator's log is inside the crust-gather host snapshot the global catch uploads, at
snapshots/opensearch/host/namespaces/cozy-opensearch-operator/..., and it shows the operator reconciling every ten seconds and then stopping at the moment the chart rewrote the CR, with nothing written for the remaining four minutes. The suite's owncatchlegs do not carry it: one selects the cluster's pods, and cozyreport keeps a pod only when it is not Running or not Ready, which a stalled operator is neither. So the evidence exists but sits behind a whole-cluster archive rather than in the job log, and #4333 adds the leg plus a watchdog that names the component a stalled reconcile sits in.Worth noting that the step is a fixed 240s poll and says so in its own comment: the re-mint happens on the operator's next reconcile and is recorded in no condition, which is why the test reads it out of the certificate. The
TODO(e2e-replace-fixed-timeouts)sitting next to it is the same problem.Checklist