Repository navigation
kafka: uninstall leaves KafkaNodePools and their brokers running #4488
Description
Describe the bug
Deleting a Kafka application does not stop its brokers. The KafkaNodePools stay in the namespace after the release is uninstalled, and the StrimziPodSets they own keep every broker pod running with its full memory request.
The pools are not part of the release manifest. Since cb7578b the migration hook creates them with kubectl apply on every install and upgrade (migration-hook.yaml#L244-L248). They carry the Helm labels and annotations, but Helm only deletes what is in the stored manifest, so helm uninstall never touches them. They have no ownerReference to the Kafka CR either, so garbage collection does not remove them when the Kafka CR goes away. The pre-delete hook deletes only KafkaTopics (hooks/delete.yaml#L81), and its Role has no permissions on kafkanodepools.
Environment
To Reproduce
- Create a Kafka application in a tenant and wait until it is Ready.
- Delete the application.
kubectl get kafkanodepools,strimzipodsets,pods -n <tenant-ns>
Expected behaviour
After the application is deleted, its node pools, pod sets and broker pods are gone. PVCs can stay, since the pools set deleteClaim: false.
Actual behaviour
The Kafka CR and the HelmRelease are gone, but the pools, the pod sets and the brokers are still there, with no deletionTimestamp. Each broker keeps its 1Gi memory request.
In e2e this adds up. The crust-gather snapshot taken when the harbor suite failed on PR #4447 (run 36012456733, 2026-09-24) still has brokers from four Kafka releases whose suites had already passed and deleted their applications: kafka-test in tenant-test (brokers created 15:15 UTC, snapshot at 17:08 UTC), and kafka-kafka-test, kafka-kafka-restore, kafka-kafka-meta-src, kafka-kafka-meta-target in tenant-root. Summed pod memory requests at that moment were 20.4, 20.4 and 23.65 GiB on nodes with about 23.9 GiB allocatable. Harbor then cannot schedule a pod whose local PV is pinned to the full node, and the suite fails its 10 minute HelmRelease wait:
Warning OutOfmemory pod/rfr-harbor-test-redis-0 Node didn't have enough resource: memory, requested: 320864256, used: 25393364992, capacity: 25664946176
Warning FailedScheduling pod/rfr-harbor-test-redis-0 0/3 nodes are available: 1 Insufficient memory, 2 node(s) didn't match PersistentVolume's node affinity.
Run 36041330090 failed harbor the same way on harbor-test-jobservice. These are the only two harbor failures I found in PR runs since 2026-09-18, so a rerun only passes when the PV lands on a node that still has room.
Additional context
The fix I would expect is for the pre-delete hook to also delete the release's KafkaNodePools (-l strimzi.io/cluster=<release>), with the matching Role rule. The other option is an ownerReference from each pool to its Kafka CR.
Not part of this bug, but in the same snapshot the kubernetes-test-latest-version TenantControlPlane had been sitting on finalizer.kamaji.clastix.io for 30 minutes, holding two 2.78Gi apiserver pods. I did not look into it.
Activity
fix(kafka): delete the release's node pools on uninstall (cozystack#4490
test(e2e): fail the suite whose pods outlive its cleanup (#4492)