Repository navigation
e2e: opensearch times out waiting for a StatefulSet roll that never starts #4259
Description
Describe the bug
The opensearch suite waits twelve minutes for the nodes StatefulSet to roll onto a new mount plan, then gives up. Two things hold at the deadline and the log says nothing about which drives the other: the live template never gained the three volumes the step asserts, and opensearch-test-nodes-0 never passed its startup probe.
Environment
- Cozystack:
main - In-tree e2e lane, run 34891318171
To Reproduce
Run the full chainsaw suite. The opensearch suite fails at verify-statefulset-rolled-onto-the-new-mount-plan after about 960s.
Expected behaviour
The operator rewrites the StatefulSet with the chart-CA mount plan and the rollout completes. Failing that, the step says which half did not happen.
Actual behaviour
The assert ran at 21:20:16 and errored at 21:32:16:
* status.(currentRevision == updateRevision): Invalid value: false: Expected value: true
* status.updatedReplicas: Required value: field not found in the input object
Its diff shows the three volume entries missing from the live object rather than merely unrolled:
-(spec.template.spec.volumes[?name == 'http-ca'].secret.secretName):
-- opensearch-test.http-ca
-(spec.template.spec.volumes[?name == 'http-cert'].secret.secretName):
-- opensearch-test-http-server
-(spec.template.spec.volumes[?name == 'http-key'].secret.secretName):
-- opensearch-test-http-server
The pod under it never came up:
14m Warning Unhealthy pod/opensearch-test-nodes-0 Startup probe failed: dial tcp 10.244.1.8:9200: connect: connection refused
13m Warning Unhealthy pod/opensearch-test-nodes-0 Readiness probe failed:
Logs
Same run, offered as a lead. Nothing ties it to the stuck rollout yet. OpenSearch logged a plaintext request arriving on its TLS listener:
[opensearch-test-nodes-0] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a203139322e3136382e3132332e3230313a393230300d0a557365722d4167656e743a206375726c2f382e352e300d0a4163636570743a202a2f2a0d0a0d0a
That hex is GET / HTTP/1.1 with Host: 192.168.123.201:9200 and User-Agent: curl/8.5.0, so something reached 9200 over plain HTTP while the listener expected TLS.
Additional context
One observation, one run. I have not established whether the missing volumes and the dead pod share a cause, whether either recurs, or whether twelve minutes is the right budget for this step.
To separate the two candidates: if the operator never rewrote the template, the StatefulSet generation and its updateRevision say so directly; if it did and the roll is blocked on a pod that will not start, the new revision carries the volumes and the pod is the whole story. This run's crust-gather snapshot holds both.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
test(opensearch): separate the mount-plan rewrite from the roll onto it
lexfrei commentedon Sep 15, 2026
The volumes are present, so that half can be closed. The diff lines quoted above are not evidence they are missing: a parenthesised JMESPath key always renders as removed whether or not it matched, and the authoritative list is the
*bullets, which name only the two status fields. #4279 splits the assert so the volume half is checked on its own, and on run 34971957589 that half passed.The readiness explanation does not survive either. Every node-pool StatefulSet carries
updateStrategy.type: OnDelete, set atpkg/builders/cluster.go:461-463in opensearch-k8s-operator v2.8.0 and the onlyUpdateStrategyassignment in the operator, so the StatefulSet controller never deletes a pod to apply a template change, Ready or not. On run 34971957589readyReplicas: 1matched, so the pod was Ready and nothing replaced it anyway. The roll belongs to the operator's own rollingRestart reconciler, which is where this should be dug next.The two runs differ on the pod: startup probe failing in the one above, Ready in this one. Same stall both times, so pod readiness is not what decides it.
One gap for whoever picks this up. The suite catch selects the cluster's pods, so the operator's logs are not in it, and the restart decision is made there. Only the host snapshot carries them.
test(opensearch): separate the mount-plan rewrite from the roll onto it
test(opensearch): separate the mount-plan rewrite from the roll onto it
lexfrei commentedon Sep 17, 2026
The stall is the operator's health gate. #4279 added the operator log to the catch, and on its earlier head the stall reproduced (run 35021437134, job 104576486900). The restart reconciler logs this every 10 seconds for the whole captured window, 25 times:
In opensearch-k8s-operator v2.8.0 this comes from
pkg/reconcilers/rollingRestart.go:182, whenCheckClusterStatusForRestartreturns not ready. That function (opensearch-gateway/services/os_data_service.go:156) returns this message only when health is not green, the yellow exception does not apply, draining is off, the cluster settings call succeeded and transientcluster.routing.allocation.enableis alreadyall(:180). Sostatus.Initializedand readiness passed, the operator reached the cluster, and health is what holds it.The yellow exception (
:269-284) needs no relocating or initializing shards, at most one unassigned shard, and a yellow.opensearch-observabilityindex.The operator doesn't log the health value, so I can't tell from this run if it was yellow or red, or which indices held it. Probably, on a single-node cluster any index with a replica stays yellow because the replica has nowhere to go, and two such shards already block the exception. Not verified. It would fit the stall being intermittent: the same test content passed on a later run, the roll took under two minutes.
Adding
_cluster/health?level=indicesto the catch would settle it next time.test(opensearch): separate the mount-plan rewrite from the roll onto it
test(opensearch): correct the rolling-restart log order in the roll-g…
lexfrei commentedon Sep 18, 2026
Found the cause, and it affects real single-replica releases too. On one data node any index that asks for a replica keeps the cluster yellow for good. The operator's restart gate at v2.8.0 opens on green, or on yellow only when
.opensearch-observabilityis the yellow index, and at OpenSearch 2.11.1 that index isauto_expand_replicas: 0-2, green on one node. So the gate never opens.Both failed runs I read (35021437134 attempt 1 and 35234103962) have the same sequence in the OpenSearch log: the bootstrap node leaves,
SecuritySSLNettyTransport ... Connection reset, and within 40 ms[security-auditlog-<date>] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]. The security plugin audits that exception: at 2.11.1.0SecuritySSLNettyTransportcallserrorHandler.logErrorright before it logs the line, andAuditLogSslExceptionHandlerturns that intoauditLog.logSSLException. The operator setsplugins.security.audit.type: internal_opensearchitself (pkg/reconcilers/configuration.go:63), so areplicas: 1release stops rolling after its first audited event, or after the first index a tenant creates with default settings.The plaintext
GET /from the first report is unrelated. It arrives after the assert has already failed, from the failure-path collector that probes LoadBalancer addresses over plain HTTP (hack/e2e-capture-dataplane.sh:1092).Upstream rewrote the gate today in opensearch-project/opensearch-k8s-operator#1569, no release carries it yet. #4317 carries it back to v2.8.0 and makes the suite force the yellow state. Not sure why the green runs had no audit index before the roll, a green run keeps no snapshot.
test(opensearch): separate the mount-plan rewrite from the roll onto it
fix(opensearch-operator): let a permanently yellow cluster restart (c…
test(opensearch): separate the mount-plan rewrite from the roll onto it
test(opensearch): separate the mount-plan rewrite from the roll onto …