fix(opensearch-operator): align DNS_BASE with the platform cluster domain - #3357

…main (#4185)

## What this PR does

The opensearch-operator resolves every in-cluster name it dials as
`<svc>.<ns>.svc.<DNS_BASE>`, and nothing set `DNS_BASE`, so it stayed on
the vendored chart default `cluster.local` while the platform runs on a
configurable domain. On a cluster that is not on `cluster.local` the
securityconfig bootstrap job waits forever on a name that never
resolves, the security configuration is never applied, and OpenSearch
nodes stay unready. That is issue #4073.

The platform now carries the domain to the operator in the
`cozystack-values` Secret, under the subchart key the vendored chart
already reads:

```yaml
opensearch-operator:
  manager:
    dnsBase: "cozy.local"
```

`charts/opensearch-operator` reads that as `.Values.manager.dnsBase` and
hands it to the manager as `DNS_BASE`, so nothing under `charts/`
changes and the package needs no patch. The value renders from
`networking.clusterDomain` rather than a literal, so a cluster that
keeps `cluster.local` keeps working, and a package rendered without the
Secret keeps the upstream `cluster.local` default. This is the route the
KEDA cluster domain already travels through the same Secret.

The whole change is one key in
`packages/core/platform/templates/apps.yaml` plus two test suites: one
pinning that `networking.clusterDomain` reaches the Secret under that
key, one pinning that the key reaches the Deployment env in the package.

### Why this is not the `global` route

Earlier revisions of this branch, and #3357 before it, emitted the
domain as `global.clusterDomain` and patched the vendored template to
read it. `global` is the one key Helm copies into every subchart, which
is why it looked right. That is also the problem. The `cozystack-values`
Secret is `valuesFrom` on every HelmRelease the operator builds unless
the PackageSource opts out with
`operator.cozystack.io/skip-cozystack-values`
(`internal/operator/package_reconciler.go:265-273`), so a new key under
`global` lands in the values of nearly every release on the cluster.

One of them broke on it. The vendored smee chart in
`packages/system/bootbox` gates on the truthiness of the whole `global`
map and then indexes `global.rbac`, so a single unrelated key under
`global` ends its render. This still reproduces on `main`:

```console
$ helm template bootbox packages/system/bootbox
# exit 0
$ helm template bootbox packages/system/bootbox --set global.clusterDomain=cozy.local
Error: cozy-bootbox/charts/smee/templates/role.yaml:4:32
  executing "cozy-bootbox/charts/smee/templates/role.yaml" at <.Values.global.rbac.type>:
    nil pointer evaluating interface {}.type
```

The subchart key reaches the operator with no patch, no change to any
other release, and nothing to harden in bootbox. I measured that rather
than reasoning about it: rendering all 170 charts under `packages/*/*`
plain and with `--set opensearch-operator.manager.dnsBase=cozy.local`,
exactly one chart's output changes, opensearch-operator, by the one
`DNS_BASE` line this PR is about. Nine other charts differ between the
two runs by exactly as much as two identical runs of the same command
differ, which is the random material Helm regenerates per render, and
thirty-five fail in both modes for reasons unrelated to the flag, with
byte-identical errors either way.

The smee hardening is still worth having, because the next key anyone
puts under `global` hits the same gate, and that gate is still on the
default branch of tinkerbell/charts. It is not carried here. It is
available as a separate change against its own package.

### On having more than one way to say the cluster domain

Review feedback on the earlier revision asked for the opposite of this:
fold the `keda:` block into `global.clusterDomain` so the platform has
one answer rather than two added a fortnight apart for the same reason.
The instinct is right and I am not doing it here, because `global` is
the channel that carries the cost. A key under it lands in the values of
nearly every release, one vendored chart already ends its render on it,
and the next chart with the same gate breaks on the day someone adds a
second key. The subchart key has no reach: the chart that declares it
reads it and everything else ignores it, which is how the KEDA domain
already travels.

That does leave the repo with several ways to tell a chart its domain,
and that complaint stands. What it avoids is a shared channel whose
blast radius grows with every new user of it. Whether the platform
should grow one general mechanism here, and which one, is worth settling
on its own rather than inside a bug fix.

### Why the stuck Job is a manual step and not a migration

The release note asks operators to delete a wedged
`<release>-securityconfig-update` Job once. On an existing OpenSearch
installation that Job is already stuck, so this is a step for everyone
the fix is for, and `packages/core/platform/images/migrations/` is the
mechanism built for exactly that. It is not in this PR because the
migration deletes Jobs in tenant namespaces, which is a wider grant and
a wider blast radius than the one key this change adds, and the fix
works without it. If it should ride the same release, it is a separate
commit and I will add it.

### Upgrade behaviour on a running cluster

Read from the operator source at v2.8.0. `DNS_BASE` is read on every
call through `helpers.ClusterDnsBase`
(`pkg/helpers/constants.go:35-41`), so the manager restart is the whole
switch. On a cluster that already runs OpenSearch clusters:

- Certificates are not re-issued. Transport and http certs are generated
only when their Secret is missing (`pkg/reconcilers/tls.go:272-295`,
`490-518`), per-node transport certs only for a pod with no entry in the
Secret (`tls.go:373-377`), the admin cert only when missing, expiring
within five days or not signed by the CA (`tls.go:176-211`). Existing
SANs keep the old suffix. Nothing in-cluster verifies them: the
securityadmin job runs `curl -k` and `securityadmin.sh -nhnv`
(`pkg/reconcilers/securityconfig.go:34,40,46`), the operator client sets
`InsecureSkipVerify` (`opensearch-gateway/services/os_client.go:77`),
Dashboards run with `opensearch.ssl.verificationMode: none`
(`pkg/builders/dashboards.go:180`).
- OpenSearch node pods do not roll, because no StatefulSet field derives
from the DNS base. Dashboards roll, since `OPENSEARCH_HOSTS` changes
(`pkg/builders/dashboards.go:46-47`).
- A securityconfig Job already stuck in its wait loop is not recreated.
The reconciler keeps an existing `<release>-securityconfig-update` Job
while its checksum annotation matches the securityconfig Secret
(`securityconfig.go:140-146`), and the target hostname is not part of
that checksum. Delete that Job once by hand. The next reconcile
recreates it against the new name, and users and roles start applying
from there.

### Known and not addressed here

The upstream `manager.dnsBase` knob stays usable. The platform writes it
through `spec.valuesFrom`, and a per-package override in
`Package.spec.components[].values` lands in `spec.values`, which Flux
merges on top of the references, so a package that needs a different
domain can still say so and win.

The fallback differs from its neighbours and that is deliberate.
`packages/apps/opensearch` falls back to `cozy.local`,
`packages/system/keda` declares `cozy.local` as its package default, and
this package stays on the upstream `cluster.local`. Inside the bundle
all three receive the same value, so nothing diverges in practice, and
`cluster.local` is the right standalone default for a Cozystack
installed over an existing kubeadm or k3s cluster.

A cluster domain that YAML reads as a number does not survive the round
trip. The platform quotes the value into the Secret, so
`networking.clusterDomain: 123` arrives as the string `123`, but the
upstream template renders it as a bare `value: 123` in the container
env, and an `EnvVar` value has to be a string. Quoting it means patching
the vendored template, which is the thing this approach removes, and an
all-numeric cluster domain is not a usable DNS name to begin with.
Before this change the key was never set from the platform, so the path
is new here even though nothing reaches it.

The vendored `opensearch-operator` Deployment template does not
reproduce from `make update` and has not for a while: a fresh pull plus
the patch differs in three places, and one of them is a live defect. The
committed `nindent 10` on `manager.extraEnv` renders invalid YAML, so
that knob cannot be used at all, while upstream's `nindent 8` works.
Nothing in the tree sets `extraEnv`, so it is latent, and the drift
predates this branch. This PR does not touch that patch at all and does
not carry a fix for it.

Nothing in CI renders a package chart with the values the platform
injects, which is why the smee breakage on the earlier approach had to
be found by hand. That is a gap in the harness.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff, which is one template key in
`packages/core/platform/templates/apps.yaml` and two test files.
`packages/core/platform/values.yaml` is untouched and gains no key, so
the hand-written platform-package table on the website is not affected.
No package, app schema, version enum, CRD, release prefix or naming
convention moves, so the Terraform provider is not reached. The
`networking.clusterDomain` key already exists and its default does not
change, so ansible-cozystack, which passes `--cluster-domain=cozy.local`
to k3s and sets only the networking CIDRs in its platform-package
template, needs nothing: the operator now follows the domain that repo
already configures. talm sets the Talos `cluster.network.dnsDomain` from
its own value and does not write the platform value either. Nothing
under `hack/`, no namespace and no package-tree layout moves, so ccp is
not reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): the operator's DNS_BASE now follows the platform cluster domain (`networking.clusterDomain`) instead of the upstream `cluster.local` default, so the securityconfig job and OpenSearch Dashboards reach the cluster on installations that use `cozy.local`. On upgrade, an existing `<release>-securityconfig-update` Job that is already stuck waiting on the old hostname is not recreated by the operator: delete it once by hand and the next reconcile recreates it against the new hostname. Existing certificates keep their SANs and keep working, nothing in-cluster verifies them.
```