e2e: opensearch suite sends its first API call before the bootstrap cluster-manager has loaded its security config #4412

Describe the bug

The opensearch chainsaw suite can fail at make-the-cluster-permanently-yellow: the very first request of the step, the PUT that creates unassignable-replica, is answered with HTTP 503. The step starts the moment the nodes StatefulSet reports readyReplicas: 1, and at that moment the elected cluster-manager can still be the operator's bootstrap pod, whose security plugin has not loaded its configuration yet.

Environment

• Cozystack: main as of b1519ac (the suite file is identical there)
• opensearch-k8s-operator 2.8.0 with the in-tree patches, OpenSearch 2.11.1
• In-tree e2e lane, run https://github.com/cozystack/cozystack/actions/runs/35840939046/job/107119797444 (attempt 1)

That run is on PR #3769, which changes two comment lines in hack/e2e-chainsaw/opensearch/chainsaw-test.yaml and nothing the suite executes. The other 13 suites in the run passed.

To Reproduce

Run the full chainsaw suite. It is timing-dependent: in the five other failed in-tree runs I checked since the step landed, the same step passed.

Expected behaviour

The step creates the index and reads the cluster back as yellow.

Actual behaviour

The StatefulSet assert finished and the step began in the same second, and the create failed 200 ms later:

10:27:52 | opensearch | verify-operator-managed-listener    | ASSERT | DONE  | apps/v1/StatefulSet @ tenant-test/opensearch-test-nodes
10:27:52 | opensearch | make-the-cluster-permanently-yellow | TRY    | BEGIN |
10:27:53 | opensearch | make-the-cluster-permanently-yellow | SCRIPT | LOG   |
    curl: (22) The requested URL returned error: 503
10:27:53 | opensearch | make-the-cluster-permanently-yellow | SCRIPT | ERROR |

The bootstrap pod, which is the cluster-manager at that point, rejects an index create in the same second, and loads its security configuration three seconds later:

[10:27:48,951][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch-test-bootstrap-0] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[10:27:52,946][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-test-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[10:27:53,056][ERROR][o.o.s.f.SecurityFilter   ] [opensearch-test-bootstrap-0] OpenSearch Security not initialized for indices:admin/create
[10:27:53,165][ERROR][o.o.s.f.SecurityFilter   ] [opensearch-test-bootstrap-0] OpenSearch Security not initialized for cluster:admin/voting_config/add_exclusions
[10:27:53,433][ERROR][o.o.s.f.SecurityFilter   ] [opensearch-test-bootstrap-0] OpenSearch Security not initialized for cluster:monitor/health
[10:27:54,318][INFO ][o.o.c.c.Coordinator      ] [opensearch-test-bootstrap-0] abdicating to {opensearch-test-nodes-0}...
[10:27:56,466][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-test-bootstrap-0] Node 'opensearch-test-bootstrap-0' initialized

The events from the catch block (read at 10:28:19, converted to absolute time) show the operator racing the same window:

~10:27:22  Warning  Unhealthy     pod/opensearch-test-nodes-0                 Readiness probe failed:
~10:27:53  Warning  Bootstrap     opensearchcluster/opensearch-test           Could not exclude opensearch-test-bootstrap-0 from voting; keeping the bootstrap pod
~10:27:53  Normal   RollingRestart opensearchcluster/opensearch-test          Starting rolling restart
~10:27:54  Normal   Bootstrap     opensearchcluster/opensearch-test           Excluded opensearch-test-bootstrap-0 from voting and removed the bootstrap pod
~10:27:57  Normal   Completed     job/opensearch-test-securityconfig-update   Job completed

Mechanism

Proven from the log:

• The step's os_api helper runs curl inside opensearch-test-nodes-0 against https://localhost:9200 (chainsaw-test.yaml:208-215), so the REST request entered on the data node, not on the bootstrap pod.
• Index creation is a cluster-manager action, and at 10:27:53 the cluster-manager was opensearch-test-bootstrap-0: it held the role until it abdicated to opensearch-test-nodes-0 at 10:27:54,318.
• The bootstrap pod's SecurityFilter rejected indices:admin/create at 10:27:53,056, 21 ms before chainsaw printed the 503. The only other index created in that second is the audit log at 10:27:53,065, and that one succeeded with cause auto(bulk api), so the rejected create is the suite's.
• The bootstrap pod finished loading its security configuration at 10:27:56,466, three seconds after the 503. The .opendistro_security index had been created at 10:27:48 and the securityconfig job completed around 10:27:57.
• The step's precondition is only status.readyReplicas: 1 on opensearch-test-nodes (chainsaw-test.yaml:178-189). Nothing in it looks at the bootstrap pod or at the cluster-manager.

Inferred, not proven:

• The readiness probe (curl -k -u admin --fail https://localhost:9200) passed on opensearch-test-nodes-0 at about 10:27:52, which I read as that node having loaded the security config itself between 10:27:48 and 10:27:52. The node's own log from that window is outside the catch block's 100-line tail, so I have no line showing it.
• Each node loads the security config on its own schedule after securityadmin writes the index, so a Ready data node says nothing about the bootstrap pod. Here the gap was about four seconds, and the suite's first request landed inside it.
• The window closes by itself: the operator's own voting exclusion needs the bootstrap pod's security to be initialized as well, failed on the same error at 10:27:53, succeeded a second later, and the bootstrap pod was removed. A request sent a few seconds later would have gone to opensearch-test-nodes-0 as cluster-manager.

How new it is

The wait (readyReplicas: 1) has been there since the suite was added in 19adffb. Until d3e228f (#4279, merged 2026-09-21) the next step after it was a JSON patch on the CR, which never talks to OpenSearch, so the window had nothing to hit. make-the-cluster-permanently-yellow is the first step to call the OpenSearch API right after the wait.

Frequency: 1 hit in 6 failed in-tree runs since #4279 merged. I checked the failed "Pull Request" runs from 2026-09-21T16:44Z onward whose E2E (in-tree) job failed: 35735604926, 35733363514, 35722827156, 35707565622, 35703080496, and this one, 35840939046. The step passed in four of the five others; 35707565622 did not reach the opensearch suite. 35733363514 failed later in the same suite, on the admin-certificate step tracked in #4332. Green runs were not counted, so this is a lower bound on how often the step runs and not a failure rate.

Relation to existing issues

• opensearch: a single-replica cluster loses quorum for good when the operator removes the bootstrap pod #4230 (closed) is the reason the bootstrap pod stays until the voting exclusion succeeds. The "Could not exclude ... keeping the bootstrap pod" event above is that fix doing its job, and it is also what keeps the uninitialized bootstrap pod as cluster-manager for a few seconds after the data node is Ready.
• opensearch: the securityconfig update job is OOMKilled at the tenant LimitRange default, so the cluster never initializes #4227 (closed) was the securityconfig job never completing. Here it completed, a few seconds late relative to the test.
• e2e: opensearch admin certificate is intermittently not re-minted under the chart CA within the step's budget #4332 and e2e: opensearch times out waiting for a StatefulSet roll that never starts #4259 are other steps of the same suite with different causes.

Possible fix (unverified)

Wait for the bootstrap handoff before the first OpenSearch API call, not only for the data node's readiness. Two candidates, neither tested:

• An error: assert that opensearch-test-bootstrap-0 no longer exists before make-the-cluster-permanently-yellow. The operator removes that pod only after the voting exclusion succeeds, and the exclusion goes through the bootstrap pod's security filter, so its absence implies the cluster-manager has moved to an initialized node. This does not hold for a topology that keeps a bootstrap pod, which this suite does not have.
• A poll of _cluster/health from opensearch-test-nodes-0 until it answers 200 before the PUT. Cluster health is served by the cluster-manager and was rejected with the same error at 10:27:53, so a 200 means the current cluster-manager has loaded the security config. _plugins/_security/health alone would not cover this, because it reports the local node.