Skip to content

kafka: uninstall leaves KafkaNodePools and their brokers running #4488

Description

Describe the bug

Deleting a Kafka application does not stop its brokers. The KafkaNodePools stay in the namespace after the release is uninstalled, and the StrimziPodSets they own keep every broker pod running with its full memory request.

The pools are not part of the release manifest. Since cb7578b the migration hook creates them with kubectl apply on every install and upgrade (migration-hook.yaml#L244-L248). They carry the Helm labels and annotations, but Helm only deletes what is in the stored manifest, so helm uninstall never touches them. They have no ownerReference to the Kafka CR either, so garbage collection does not remove them when the Kafka CR goes away. The pre-delete hook deletes only KafkaTopics (hooks/delete.yaml#L81), and its Role has no permissions on kafkanodepools.

Environment

  • Cozystack: main at 4a2dea2, and every build since cb7578b
  • Provider: seen in the in-tree e2e sandbox

To Reproduce

  1. Create a Kafka application in a tenant and wait until it is Ready.
  2. Delete the application.
  3. kubectl get kafkanodepools,strimzipodsets,pods -n <tenant-ns>

Expected behaviour

After the application is deleted, its node pools, pod sets and broker pods are gone. PVCs can stay, since the pools set deleteClaim: false.

Actual behaviour

The Kafka CR and the HelmRelease are gone, but the pools, the pod sets and the brokers are still there, with no deletionTimestamp. Each broker keeps its 1Gi memory request.

In e2e this adds up. The crust-gather snapshot taken when the harbor suite failed on PR #4447 (run 36012456733, 2026-09-24) still has brokers from four Kafka releases whose suites had already passed and deleted their applications: kafka-test in tenant-test (brokers created 15:15 UTC, snapshot at 17:08 UTC), and kafka-kafka-test, kafka-kafka-restore, kafka-kafka-meta-src, kafka-kafka-meta-target in tenant-root. Summed pod memory requests at that moment were 20.4, 20.4 and 23.65 GiB on nodes with about 23.9 GiB allocatable. Harbor then cannot schedule a pod whose local PV is pinned to the full node, and the suite fails its 10 minute HelmRelease wait:

Warning  OutOfmemory       pod/rfr-harbor-test-redis-0  Node didn't have enough resource: memory, requested: 320864256, used: 25393364992, capacity: 25664946176
Warning  FailedScheduling  pod/rfr-harbor-test-redis-0  0/3 nodes are available: 1 Insufficient memory, 2 node(s) didn't match PersistentVolume's node affinity.

Run 36041330090 failed harbor the same way on harbor-test-jobservice. These are the only two harbor failures I found in PR runs since 2026-09-18, so a rerun only passes when the PV lands on a node that still has room.

Additional context

The fix I would expect is for the pre-delete hook to also delete the release's KafkaNodePools (-l strimzi.io/cluster=<release>), with the matching Role rule. The other option is an ownerReference from each pool to its Kafka CR.

Not part of this bug, but in the same snapshot the kubernetes-test-latest-version TenantControlPlane had been sitting on finalizer.kamaji.clastix.io for 30 minutes, holding two 2.78Gi apiserver pods. I did not look into it.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 26, 2026
  2. added a commit that references this issue on Sep 26, 2026
    b113cc7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugtriage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions