fix(kafka): pass the container limits to the Kafka JVMs - #4865

What this PR does

The Kafka JVMs we ship size themselves from host memory instead of their container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry JDK 17.0.15, which reads /proc/cgroups to decide whether it runs in a container. Kernels 6.12 and newer built without CONFIG_MEMCG_V1 no longer list the memory controller there, so the JVM turns container support off. The JDK fix is JDK-8347811, backported to 17.0.20 and 21.0.10.

In CI this shows up in two places. The cluster operator gets a 25 GiB max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the kernel OOM-killed a Java process at about 500M, the limit the cluster operator shares with the topic and user operators. Brokers and controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi asks the same JDK for the container memory and takes half of what it reports.

This PR passes the limits to every Kafka JVM explicitly:

• the cluster operator gets -XX:MaxRAM and -XX:ActiveProcessorCount from its own limits through the downward API;
• brokers, controllers, the topic and user operators and the drain-time Cruise Control get JAVA_TOOL_OPTIONS with -XX:MaxRAM set to their memory limit, plus -XX:ActiveProcessorCount where the chart sets a CPU limit. The chart sets one only for brokers and controllers. A LimitRange default on the entity operators or Cruise Control is invisible to the chart, and Strimzi template env takes no resourceFieldRef, so those JVMs still count host CPUs.

Every java in a container reads JAVA_TOOL_OPTIONS, including the heap probe Strimzi runs before starting Kafka. So Strimzi's own heap rules and Kafka's G1 defaults apply to the real memory limit, and the chart does no heap arithmetic. jvmOptions with -XX would not work here: Strimzi turns those into KAFKA_JVM_PERFORMANCE_OPTS, and Kafka skips its G1 settings when that is set. Each JVM now prints a "Picked up JAVA_TOOL_OPTIONS" line on start.

In a kind cluster on such a kernel the operator then runs Serial GC with a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka, against 440Mi before. A broker with a 1Gi limit and one CPU gets the same -Xmx518982205 that Strimzi computes when the JDK sees the limit, with G1 and one concurrent GC thread.

Existing Kafka clusters roll once on upgrade, Talos ones included, because the pod environment changes. Their heap does not change on a kernel where the JDK already sees the limit.

The chart has no values key for JVM options or heap, so there is no user setting this could override.

The Kafka migration hook now applies with --openapi-patch=false. Its container runs under a tenant LimitRange default of 128Mi, and a plain kubectl apply of an existing pool downloads the cluster's whole OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade in such a tenant failed on an OOM-killed hook. With the flag kubectl builds the same merge patch from the last-applied annotation and peaks at about 35Mi. A larger memory limit would count against the tenant quota, and a server-side apply would stop removing fields that kubectl create set on the kafka pool. Other Jobs without resources are tracked in #4868.

The operator OOMKill is what triggered the e2e failure in #4855. The other half is a Strimzi race. After a restart, the PodSet controller drops the ADD event of a PodSet that has no pods yet and only picks it up at the 5-minute resync. That one is upstream and is described in the issue. The other JVMs we ship are tracked in #4864.

Talos nodes are not affected, the Talos kernel keeps CONFIG_MEMCG_V1=y. Generic-variant clusters on newer distribution kernels are. CI is affected even though it runs Talos, because there the Talos nodes are containers on the runner's Ubuntu 6.17 kernel.

Closes #4855

Screenshots

Not a UI change.

Downstream repositories

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade.

Summary by CodeRabbit

• New Features

  • Kafka components now size JVM memory and processor settings based on their configured resource limits, including during migrations.
  • The Kafka cluster operator now receives container resource limits for JVM sizing.

• Bug Fixes

  • Kafka migration operations avoid OpenAPI patch lookups that could use excessive memory.

• Tests

  • Added coverage for JVM sizing, resource limits, and migration behavior.