opensearch-operator: every call to a cluster's HTTP API can wait forever, on a single worker #4337

Every call the operator makes to a cluster's HTTP API can wait forever, and it runs one worker, so one unanswered request stops all reconciliation.

Measured in the v2.8.0 sources this image builds from, opensearch-operator/opensearch-gateway/services/os_client.go: 14 calls, every one req.Do(context.Background(), ...), no deadline and no cancellation. NewOsClusterClient builds its own transport at lines 76-82, &http.Transport{TLSClientConfig, DisableKeepAlives, MaxIdleConns}, and a fresh http.Transport carries no timeouts at all: no dial, no TLS handshake, no response header. http.DefaultTransport would have had the first two. The config sets no Client, so there is no http.Client.Timeout either.

The request types do carry a Timeout, and it is the server-side query parameter rather than a client deadline: opensearch-go v1.1.0 opensearchapi/api.cluster.health.go:120-121 puts it in params["timeout"]. It bounds how long the server waits before answering, which helps only if something answers. GetHealth does not set it anyway.

The measurement is in #4332. In run 35333289958 the operator logged Reconciling OpenSearchCluster every ten seconds and stopped at 10:56:35.706, with nothing until collection at 11:01:11, on a complete 144-line log with no container restart. The last line before the silence is the cluster reconciler excluding the bootstrap pod from voting, which goes to the HTTP API, and the pod it was talking to was NotReady.

The discriminator is in those runs too. In both red runs the cleanup afterwards hangs for exactly five minutes and fails with OpenSearchCluster/tenant-test/opensearch-test termination timeout: context deadline exceeded, same finalizer, same worker. In the green run it takes seconds.

This is also why the symptom keeps moving. A blocked worker surfaces as whatever the suite is waiting for: the roll in #4259, the admin certificate re-mint in #4332, the StatefulSet still on the pre-move http-cert volume with both Certificates Ready. #4317 removed one place it showed up, not the reason a pass can stop.

Fix is a deadline per call, or timeouts on the transport, or both, as a patch against os_client.go next to the three we already carry. Upstream is the better home for it. The watchdog in #4333 is worth having either way, since it turns the next occurrence into a name instead of a silence, but it does not remove the unbounded wait.