Repository navigation
Deleting a tenant with a Kafka app hangs the namespace in Terminating (KafkaTopic strimzi.io/topic-operator finalizer) #3793
Description
Activity
Good write-up. This is the same class of "operator torn down before its finalizer-owning CRs" bug that has already been fixed for several other apps in the repo, and the mechanism is well understood.
Root cause
The Kafka chart's
topics.yamlcreatesKafkaTopicCRs with nofinalizersfield [1] — Strimzi's entity-operator adds thestrimzi.io/topic-operatorfinalizer at runtime when it takes ownership. When the Kafka app (and the entity-operator pod) is torn down first, nothing remains to drain that finalizer, so theKafkaTopicobjects block namespace GC indefinitely. This was already observed as a side-effect in PR #2934, which noted that a CrashLoopBackOff entity-operator leavesKafkaTopics stuck with the same finalizer.No pre-delete hook exists for Kafka today
The
packages/apps/kafka/templatesdirectory contains onlykafka.yaml,topics.yaml, and monitoring/config templates — there is nodelete.yamlor any pre-delete hook [2]. The tenant-level cleanup-job.yaml deletes HelmReleases in the namespace with a 90s bounded wait, but that only removes the Kafka HelmRelease — it has no authority overKafkaTopicCRs or their Strimzi finalizers.Existing patterns to follow
The repo already has two merged fixes for the identical problem class:
- PR #3078 — the
packages/apps/kuberneteschart'sdelete.yamldeletes theTenantControlPlanewith--wait=true --timeout=120sto let Kamaji drain its own finalizer, then as a backstop patchesmetadata.finalizers: nullon the datastore-config secret if it still exists. - PR #3598 — the tenant pre-delete hook with bounded waits,
activeDeadlineSeconds: 270,backoffLimit: 0, andrestartPolicy: Never, all sized to fit inside helm-controller's 300s uninstall budget.
Suggested fix direction
Add a Helm
pre-deletehook Job to thekafkachart (mirroring the kubernetes app'sdelete.yaml) that:- Deletes all
KafkaTopicCRs in the namespace with--wait=true --timeout=<N>s— while the entity-operator is still alive to process the deletion and drain its finalizer. - As a backstop after the timeout, patches
metadata.finalizers: nullon any survivingKafkaTopicCRs — safe because theKafkaCR and its operator are being torn down in the same release. - Carries
activeDeadlineSecondsset below the HelmRelease uninstall budget,backoffLimit: 0, andrestartPolicy: Never, following the same arithmetic used in the tenant and kubernetes hooks.
Your manual workaround (patching
finalizers: null) is correct in the interim.To reply, just mention @dosu.
Docs are dead. Just use Dosu.
- PR #3078 — the
Reproduced on Cozystack with Strimzi 0.45. Helm uninstall removes the topic operator together with the KafkaTopics, so a topic still carrying the
strimzi.io/topic-operatorfinalizer blocks deletion, and the next install with the same release name times out on the leftover topic.Opened #3938 with a pre-delete hook that deletes only release-labelled KafkaTopics while the operator is still running, without forcing finalizer removal. Render and unit tests pass, and the kafka e2e suite gains an absence assert for the release topics after deletion.
Still reproduces on
mainatb222cb658, and there is a second entry point worth recording: deleting the app hits it, not only deleting the tenant.From the
cozyreportof anE2E (in-tree)run,kafkasuite, stepverify-pvc-reclaim-on-delete:16:35:32 DELETE RUN apps.cozystack.io/v1alpha1/Kafka @ tenant-test/test 16:40:32 DELETE ERROR context deadline exceededFive minutes, the operation's whole budget, on deleting the Kafka CR itself. No tenant is being removed here.
What the snapshot holds at that moment:
KafkaTopic kafka-test-test-orders deletionTimestamp=2026-09-14T16:38:16Z finalizers=[strimzi.io/topic-operator] KafkaTopic kafka-test-test-results deletionTimestamp=2026-09-14T16:38:16Z finalizers=[strimzi.io/topic-operator] HelmRelease kafka-test deletionTimestamp set finalizers=[finalizers.fluxcd.io]and not one Kafka pod left in the namespace — the entity operator that owns that finalizer is already gone. So the ordering is the same one #3938 fixes, reached from the app side: the release tears down the topic operator, the topics keep a finalizer only it can clear, and the HelmRelease never finalizes behind them.
Two things this adds.
The e2e suite already catches it.
hack/e2e-chainsaw/kafka/chainsaw-test.yamlfails on this in ordinary runs, so the defect is not waiting on someone to reproduce it by hand — it is one of the failures authors of unrelated PRs are currently working around. #3938 is open since 9 Sep and the hook is not onmain, so every run that reaches that step spends five minutes proving the same thing.And the app-deletion path may want its own assertion. #3938 adds an absence assert for the release topics after deletion, which covers this if the suite deletes the app; worth confirming that is the path the new assert exercises, since the tenant path and the app path tear things down in a different order.
Reacted by europrinter- added and removed
on Sep 23, 2026 - added a commit that references this issue
on Sep 25, 2026
Summary
Deleting a tenant that has a Kafka app leaves the tenant namespace stuck in
Terminatingindefinitely. TheKafkaTopicCR keeps the finalizerstrimzi.io/topic-operator, but the Strimzi topic-operator is torn down together with the Kafka app, so nothing ever removes the finalizer and the namespace never finalizes.Environment
v1.6.1-rc.1Steps to reproduce
orders).Actual result
The tenant namespace hangs in
Terminating:The
KafkaTopic(e.g.kafka-kafka-orders) still carriesfinalizers: [strimzi.io/topic-operator]. The Strimzi topic-operator has already been deleted together with the Kafka app, so no controller is left to drain that finalizer. The namespace stays inTerminatingforever.Reproduced on every tenant that had a Kafka app (multiple tenants stuck the same way).
Expected result
Deleting the tenant / Kafka app should clean up
KafkaTopicresources (or drain their finalizers) so the namespace terminates cleanly, without manual finalizer surgery.Workaround
Remove the finalizer manually, then the namespace finalizes within seconds:
Suggested fix
Order the teardown so the Strimzi topic-operator outlives its
KafkaTopicresources (drain thestrimzi.io/topic-operatorfinalizer before the operator itself is removed), or have the Kafka app / tenant deletion explicitly deleteKafkaTopicCRs and wait for their finalizers to clear before removing the operator.