Repository navigation
e2e(kafka): Strimzi operator OOMKilled mid-create leaves the KRaft controller PodSet unreconciled #4855
Description
Activity
I found both mechanisms, and they are two separate bugs. The OOM comes from the JVM not seeing the container limit. The ignored PodSet is a Strimzi race that any operator restart triggers.
The operator went over 512Mi because its JVM sized itself from the host. The runner kernel
6.17.0-1020-oracleis built withoutCONFIG_MEMCG_V1. I checked the config shipped in itslinux-buildinfopackage. Since 6.12, such a kernel drops thememoryline from/proc/cgroups, and the JDK 17.0.15 in the Strimzi 0.45.1 image still reads that file to decide whether it runs in a container. With the line missing it turns container support off. On the 128 GiB runner that means G1 with a 25 GiB max heap and a 2 GiB initial heap inside a 512Mi limit, so the heap was never capped at 100Mi. This is JDK-8347811, fixed in 17.0.20 and 21.0.10.I reproduced the detection failure in kind, on a host whose kernel also lacks
CONFIG_MEMCG_V1. The JVM logs "controller memory is not enabled" and reportsHeap Max Capacity: 5696Mon a 28 GiB host. With the CI host size emulated (-XX:MaxRAM=126g -XX:ActiveProcessorCount=32), Native Memory Tracking shows 521 MB committed right after the Kafka became Ready, already above the limit. With-XX:MaxRAM=512m -XX:ActiveProcessorCount=1the JVM picks Serial GC and a 78M max heap. The container then peaks at 249Mi for the same Kafka creation, against 440Mi without the flags.Talos nodes are fine, the Talos kernel has
CONFIG_MEMCG_V1=y. Generic distributions on 6.12+ kernels without it are affected. So is CI, because there Talos runs in containers on the runner's Ubuntu kernel.The controller PodSet was ignored because of a startup race in Strimzi's PodSet controller, and it does not need an OOM. On start the PodSet informer is started before the Kafka informer, and the ADD handler looks up the owning Kafka in the Kafka informer cache. While that cache is still empty the lookup fails and the event is dropped, logged only at DEBUG. A PodSet with pods gets re-enqueued by pod events once the informers sync, which is why the broker PodSet recovered. A PodSet without pods waits for the informer resync. That is 5 minutes, the same as the suite's wait, so it would have recovered at about 20:29, a minute after the collection.
I reproduced the race three times out of three in kind. I stopped the operator, deleted the controller pool's pods and started it again with DEBUG logging. Every time the ADD was dropped at startup ("does not belong to a custom resource matching the selector") and the pods came back after 305 to 309 seconds, right after the resync. The code is the same on Strimzi
main, so an operator upgrade would not fix it. I'm reporting it upstream.The fix PR passes the limits to the JVM explicitly. That removes the restart that triggered the race here.
In one PR e2e run the kafka suite failed because the KRaft controller pods were never created. The Strimzi cluster operator was OOMKilled two seconds after it created both StrimziPodSets, and after the restart it never reconciled the controller one.
Run: https://github.com/cozystack/cozystack/actions/runs/38077365379/job/114291191008 (PR #4657, which only changes
hack/cozyreport.sh). The suite'swait-readystep timed out onKafkaTopic tenant-test/kafka-test-test-results(20:23:37 to 20:28:37 UTC).Timeline from the crust-gather snapshot of the failed suite (
snapshots/kafkain thecozyreportartifact, taken at 20:28:40 UTC):Kafka tenant-test/kafka-testand issues certificates for thec-1b874378controller nodes 2, 3 and 4.kafka-test-b-1b874378(brokers 0, 1) andkafka-test-c-1b874378(controllers 2, 3, 4).reason: OOMKilled, limit 512Mi,-XX:MaxRAMPercentage=20). It had been running since 19:25 and served the earlier suites.Reconciliation #1(watch) Kafka(tenant-test/kafka-test): Reconciliation is in progress, once a minute.At collection time:
kafka-test-c-1b874378StrimziPodSet exists with the three pods in its spec and has nostatusat all. Both operator logs, before and after the restart, have zero lines forStrimziPodSet(tenant-test/kafka-test-c-1b874378), while the broker PodSet is reconciled more than twenty times after the restart.kafka-test-c-*pod exists. Their PVCs exist and stayPendingwithWaitForFirstConsumer.kafka-test-kafka-brokersexists, and its EndpointSlice has onlykafka-test-b-1b874378-0and-1, both ready.java.net.UnknownHostException: kafka-test-c-1b874378-N.kafka-test-kafka-brokers...fromRaftManager, fail liveness and go into BackOff. The Kafka staysNotReady / Creating.What I have not established: why the PodSet controller never picked up the controller PodSet after the restart, when the informer's initial list should have queued it, and whether the OOM is needed for that at all. I also don't know what took the operator over 512Mi. The heap is capped at about 100Mi, so it was not the Java heap. This is one run, and I haven't found another with the same signature.