On kernels 6.12 and newer built without CONFIG_MEMCG_V1, a JDK older than 17.0.20 or 21.0.10 does not see the container limits and sizes itself from the host (JDK-8347811). It still reads /proc/cgroups to find out whether the memory controller is enabled, and on those kernels that file no longer lists it. The heap ceiling then becomes a share of host RAM and the GC and compiler threads follow the host CPU count. Talos is not affected, its kernel keeps CONFIG_MEMCG_V1=y. The generic variant on such kernels is affected, and so is CI, where Talos runs in containers on an Ubuntu 6.17 kernel without it.
#4855 hit this in the Kafka area, and the PR for it covers the Strimzi operator, the brokers and controllers, the entity operators and the drain-time Cruise Control. This issue tracks the other JVMs we ship. I checked each image by running it under --memory 512m --cpus 1 on a kernel without CONFIG_MEMCG_V1 and reading MaxHeapSize from -XX:+PrintFlagsFinal. A JVM that sees the limit reports 128M. One that does not reports 7.4G on that 28 GiB host.
| Component |
Image |
JDK |
Sees the limit |
Heap set explicitly |
Effect |
| Keycloak |
quay.io/keycloak/keycloak:26.6.3 |
21.0.11 |
yes |
yes, JAVA_OPTS_KC_HEAP from cozy-lib.resources.javaHeap |
none |
| keycloak-configure jobs |
quay.io/keycloak/keycloak:26.5.2 |
21.0.10 |
yes |
no |
none |
| OpenSearch nodes |
opensearchproject/opensearch 1.3.20 / 2.11.1 / 3.0.0 |
11.0.25 / 17.0.8 / 21.0.7 |
no |
yes, the operator sets -Xmx and -Xms to half the memory request |
heap is fine, GC and compiler threads follow the host CPU count |
| OpenSearch securityadmin update job |
same images |
same |
no |
no |
heap ceiling is a quarter of host RAM inside a 512Mi limit, and the job has backoffLimit: 0, so an OOM kill leaves the security index unwritten |
| LINSTOR controller |
ghcr.io/cozystack/cozystack/piraeus-server:v1.6.0 |
17.0.19 |
no |
yes, -Xmx8G in the start script |
none, the container has no memory limit to detect |
| LINSTOR satellite |
same image |
17.0.19 |
no |
yes, -Xmx2G in the start script |
detection does not change the heap |
The OpenSearch images are the ones that need a change, since no OpenSearch version in files/versions.yaml ships a fixed JDK. The securityadmin job is the real risk: its heap has no ceiling the container can enforce, and the only retry is deleting the release. Passing -XX:MaxRAM with the job's limit, or an explicit -Xmx, would fix it. For the nodes, -XX:ActiveProcessorCount set to the CPU limit would bring the thread counts back to what a working JDK picks.
The LINSTOR image is fine as long as the heap stays explicit. A JDK of 17.0.20, 21.0.10 or later in the piraeus-server image removes the question.
One unrelated thing I noticed: packages/system/linstor/templates/satellites-cozy.yaml describes the satellite as "a Go binary with a bounded working set". It is a JVM, started with -Xmx2G against a 2Gi container limit, so the heap ceiling alone fills the limit before any off-heap memory.
On kernels 6.12 and newer built without
CONFIG_MEMCG_V1, a JDK older than 17.0.20 or 21.0.10 does not see the container limits and sizes itself from the host (JDK-8347811). It still reads/proc/cgroupsto find out whether the memory controller is enabled, and on those kernels that file no longer lists it. The heap ceiling then becomes a share of host RAM and the GC and compiler threads follow the host CPU count. Talos is not affected, its kernel keepsCONFIG_MEMCG_V1=y. The generic variant on such kernels is affected, and so is CI, where Talos runs in containers on an Ubuntu 6.17 kernel without it.#4855 hit this in the Kafka area, and the PR for it covers the Strimzi operator, the brokers and controllers, the entity operators and the drain-time Cruise Control. This issue tracks the other JVMs we ship. I checked each image by running it under
--memory 512m --cpus 1on a kernel withoutCONFIG_MEMCG_V1and readingMaxHeapSizefrom-XX:+PrintFlagsFinal. A JVM that sees the limit reports 128M. One that does not reports 7.4G on that 28 GiB host.quay.io/keycloak/keycloak:26.6.3JAVA_OPTS_KC_HEAPfromcozy-lib.resources.javaHeapquay.io/keycloak/keycloak:26.5.2opensearchproject/opensearch1.3.20 / 2.11.1 / 3.0.0-Xmxand-Xmsto half the memory requestbackoffLimit: 0, so an OOM kill leaves the security index unwrittenghcr.io/cozystack/cozystack/piraeus-server:v1.6.0-Xmx8Gin the start script-Xmx2Gin the start scriptThe OpenSearch images are the ones that need a change, since no OpenSearch version in
files/versions.yamlships a fixed JDK. The securityadmin job is the real risk: its heap has no ceiling the container can enforce, and the only retry is deleting the release. Passing-XX:MaxRAMwith the job's limit, or an explicit-Xmx, would fix it. For the nodes,-XX:ActiveProcessorCountset to the CPU limit would bring the thread counts back to what a working JDK picks.The LINSTOR image is fine as long as the heap stays explicit. A JDK of 17.0.20, 21.0.10 or later in the piraeus-server image removes the question.
One unrelated thing I noticed:
packages/system/linstor/templates/satellites-cozy.yamldescribes the satellite as "a Go binary with a bounded working set". It is a JVM, started with-Xmx2Gagainst a 2Gi container limit, so the heap ceiling alone fills the limit before any off-heap memory.