Repository navigation
JVMs in the OpenSearch images ignore the container limits on kernels without CONFIG_MEMCG_V1 #4864
Description
On kernels 6.12 and newer built without CONFIG_MEMCG_V1, a JDK older than 17.0.20 or 21.0.10 does not see the container limits and sizes itself from the host (JDK-8347811). It still reads /proc/cgroups to find out whether the memory controller is enabled, and on those kernels that file no longer lists it. The heap ceiling then becomes a share of host RAM and the GC and compiler threads follow the host CPU count. Talos is not affected, its kernel keeps CONFIG_MEMCG_V1=y. The generic variant on such kernels is affected, and so is CI, where Talos runs in containers on an Ubuntu 6.17 kernel without it.
#4855 hit this in the Kafka area, and the PR for it covers the Strimzi operator, the brokers and controllers, the entity operators and the drain-time Cruise Control. This issue tracks the other JVMs we ship. I checked each image by running it under --memory 512m --cpus 1 on a kernel without CONFIG_MEMCG_V1 and reading MaxHeapSize from -XX:+PrintFlagsFinal. A JVM that sees the limit reports 128M. One that does not reports 7.4G on that 28 GiB host.
| Component | Image | JDK | Sees the limit | Heap set explicitly | Effect |
|---|---|---|---|---|---|
| Keycloak | quay.io/keycloak/keycloak:26.6.3 |
21.0.11 | yes | yes, JAVA_OPTS_KC_HEAP from cozy-lib.resources.javaHeap |
none |
| keycloak-configure jobs | quay.io/keycloak/keycloak:26.5.2 |
21.0.10 | yes | no | none |
| OpenSearch nodes | opensearchproject/opensearch 1.3.20 / 2.11.1 / 3.0.0 |
11.0.25 / 17.0.8 / 21.0.7 | no | yes, the operator sets -Xmx and -Xms to half the memory request |
heap is fine, GC and compiler threads follow the host CPU count |
| OpenSearch securityadmin update job | same images | same | no | no | heap ceiling is a quarter of host RAM inside a 512Mi limit, and the job has backoffLimit: 0, so an OOM kill leaves the security index unwritten |
| LINSTOR controller | ghcr.io/cozystack/cozystack/piraeus-server:v1.6.0 |
17.0.19 | no | yes, -Xmx8G in the start script |
none, the container has no memory limit to detect |
| LINSTOR satellite | same image | 17.0.19 | no | yes, -Xmx2G in the start script |
detection does not change the heap |
The OpenSearch images are the ones that need a change, since no OpenSearch version in files/versions.yaml ships a fixed JDK. The securityadmin job is the real risk: its heap has no ceiling the container can enforce, and the only retry is deleting the release. Passing -XX:MaxRAM with the job's limit, or an explicit -Xmx, would fix it. For the nodes, -XX:ActiveProcessorCount set to the CPU limit would bring the thread counts back to what a working JDK picks.
The LINSTOR image is fine as long as the heap stays explicit. A JDK of 17.0.20, 21.0.10 or later in the piraeus-server image removes the question.
One unrelated thing I noticed: packages/system/linstor/templates/satellites-cozy.yaml describes the satellite as "a Go binary with a bounded working set". It is a JVM, started with -Xmx2G against a 2Gi container limit, so the heap ceiling alone fills the limit before any off-heap memory.
Activity
fix(kafka): pass the container limits to the Kafka JVMs (#4865)