opensearch: the securityconfig update job is OOMKilled at the tenant LimitRange default, so the cluster never initializes #4227

Describe the bug

The securityconfig-update Job that the opensearch-k8s-operator runs for a managed OpenSearch release has no resource requests or limits, neither from the chart nor from the operator. In a tenant namespace the tenant-range-limits LimitRange therefore supplies its own default of 128Mi, and securityadmin is killed by the OOM killer a few seconds after it connects.

Nothing then writes the .opendistro_security index, so every node stays stuck reporting Not yet initialized (you may need to run securityadmin) and its readiness probe never passes. The StatefulSet never reports a ready replica and the release never becomes usable.

This is currently red in every CI run that executes the chainsaw app suite. hack/e2e-chainsaw/opensearch/chainsaw-test.yaml waits 12 minutes for status.readyReplicas: 1 on opensearch-test-nodes in the verify-operator-managed-listener step, and that wait is the whole 748s runtime of the failing test.

Environment

• Cozystack: main, in-tree e2e lane (container sandbox), runs from 2026-09-11 onward
• Provider: GitHub Actions e2e sandbox

To Reproduce

1. Install a managed OpenSearch release into a tenant namespace that carries the default tenant-range-limits LimitRange.
2. Wait for the operator to create the <release>-securityconfig-update Job.
3. kubectl -n <tenant-ns> get job <release>-securityconfig-update -o yaml

Expected behaviour

securityadmin gets enough memory to complete, the security index is written, and the node pods reach ready.

Actual behaviour

The Job's pod is OOMKilled on its only attempt and the Job gives up with BackoffLimitExceeded. The cluster CR sits at phase: RUNNING, health: unknown forever, and the only event on it is Starting securityconfig update job.

The container's applied resources match the namespace LimitRange defaults field for field, which is what identifies the source: neither the chart nor the operator sets anything.

# LimitRange/tenant-range-limits in the tenant namespace
spec:
  limits:
  - default:
      cpu: 250m
      ephemeral-storage: 2Gi
      memory: 128Mi
    defaultRequest:
      cpu: 25m
      ephemeral-storage: 50Mi
      memory: 128Mi
    type: Container

packages/apps/opensearch/templates/opensearch.yaml renders spec.security.config with securityConfigSecret and adminCredentialsSecret only. updateJob appears in no template in the package, and the operator CRD does accept spec.security.config.updateJob.resources.

Logs

Pod status of the Job's only pod:

state:
  terminated:
    exitCode: 137
    reason: OOMKilled
    startedAt: 2026-09-12T20:12:25Z
    finishedAt: 2026-09-12T20:14:38Z
resources:
  limits:
    cpu: 250m
    ephemeral-storage: 2Gi
    memory: 128Mi
  requests:
    cpu: 25m
    ephemeral-storage: 50Mi
    memory: 128Mi

Everything the updater container managed to log before it was killed:

Waiting to connect to the cluster
OpenSearch Security not initialized.
Security Admin v7
Will connect to opensearch-test.tenant-test.svc.cozy.local:9200 ... done

Job status:

conditions:
- type: Failed
  status: "True"
  reason: BackoffLimitExceeded
  message: Job has reached the specified backoff limit
failed: 1

What the node pods report for as long as the suite waits:

[ERROR][o.o.s.c.ConfigurationLoaderSecurity7] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[ERROR][o.o.s.a.BackendRegistry  ] Not yet initialized (you may need to run securityadmin)

The chainsaw assertion that times out:

--- FAIL: chainsaw/opensearch (748.82s)
apps/v1/StatefulSet/tenant-test/opensearch-test-nodes
* status.readyReplicas: Required value: field not found in the input object

Additional context

The failure is deterministic, not a flake. Six runs across four unrelated pull requests, none of which touches OpenSearch, fail on this one test and on nothing else, all within one and a half seconds of the same 12 minute wait:

I pulled the cozyreport artifact from two of those runs, on two different pull requests, and both carry the same exitCode: 137 with the same 128Mi limit, so the OOM is not specific to one host or one scheduling accident.

The suite itself arrived with #2682, merged 2026-09-11, which added hack/e2e-chainsaw/opensearch/chainsaw-test.yaml. Green e2e results dated after that merge come from runs where test selection skipped the chainsaw lane and only test-openapi ran, so as far as I can tell this suite has never passed. The chart defect it exposes is older than the suite: nothing in the package has ever set resources for the update job.

Two problems sit on top of each other here. The chart one is that spec.security.config.updateJob.resources is unset, which hands a JVM 128Mi in any namespace with the tenant LimitRange. The reporting one is that a failed security bootstrap surfaces as a 12 minute readiness timeout with no mention of the Job, which is a slow and indirect way to learn that one pod was killed for memory 10 minutes earlier.

Same shape as #4220 for the barman-cloud sidecar, and the same family as #4003 and #3519.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs