Repository navigation
kafka: the pre-delete hook cannot reach the apiserver, so every Kafka release now refuses to uninstall #4276
Description
Describe the bug
The Kafka pre-delete hook added in #3938 cannot reach the apiserver. kubectl inside it falls back to http://localhost:8080, every call is refused, the script takes its catch-all branch and exits 1, and a non-zero pre-delete hook returns from Uninstall.Run before any release resource is removed. helm-controller then retries the uninstall forever: the e2e run below shows six kafka-test-pre-delete-* pods in five minutes, each one dying the same way.
The net effect is the opposite of what the hook was written for. #3793 is a Kafka release that will not finish deleting; this makes every Kafka release refuse to delete, whether or not a topic ever held the finalizer.
Environment
- Cozystack:
main, ata19fd0274(the merge of fix(kafka): delete release topics before the topic operator is removed #3938) - The e2e lane,
E2E (fork)run 34963349053
To Reproduce
Install a Kafka application and delete it. The hook runs before anything is removed, fails, and the release stays.
Expected behaviour
The hook lists and deletes the release's KafkaTopics through the in-cluster credentials of the ServiceAccount it declares, and the uninstall proceeds.
Actual behaviour
[pod/kafka-test-pre-delete-bwlq7/cleanup] Deleting KafkaTopics for kafka-test before the topic-operator is removed
[pod/kafka-test-pre-delete-bwlq7/cleanup] E0915 12:10:05.495340 7 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=20s\": dial tcp [::1]:8080: connect: connection refused"
[pod/kafka-test-pre-delete-bwlq7/cleanup] The connection to the server localhost:8080 was refused - did you specify the right host or port?
[pod/kafka-test-pre-delete-bwlq7/cleanup] ERROR: deleting KafkaTopics failed: ...
Two suites go red as a result:
--- FAIL: chainsaw/kafka (646.93s)
--- FAIL: chainsaw/kafka-3-metadata-roundtrip (233.26s)
The first fails at verify-pvc-reclaim-on-delete | DELETE on apps.cozystack.io/v1alpha1/Kafka @ tenant-test/test, which is the uninstall the hook is blocking.
Additional context
localhost:8080 is what kubectl uses when it found neither a kubeconfig nor in-cluster credentials, so this is not RBAC. Insufficient RBAC answers 403 and would have named the verb; the hook never got as far as a request to the cluster.
What the root cause is not, as far as the tree shows. I compared the hook against the two that work: packages/apps/kubernetes/templates/delete.yaml, which is also pre-delete, and packages/apps/harbor/templates/hooks/cleanup.yaml, which is post-delete. All three carry the same clastix/kubectl:v1.32 image, the same runAsUser: 65534 with runAsNonRoot and readOnlyRootFilesystem, the same HOME=/tmp, the same policy.cozystack.io/allow-to-apiserver: "true" pod label, the same serviceAccountName naming, and the same hook weights, 0 for the ServiceAccount, 5 for the Role and RoleBinding, 10 for the Job. Nothing in the tree sets automountServiceAccountToken: false on any of it. Whatever separates them is not visible in the manifests, so it needs a cluster to find.
Worth recording about how this reached main: the e2e lane never ran on #3938 while it was open. Its workflows sat at action_required for the whole life of the PR because it came from a fork, so the only checks that ever reported were the labelers and DCO. The suite ran for the first time after a maintainer authorised it, and it reported failure at 12:21, twenty minutes after the merge at 12:00.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
IvanHunters commentedon Sep 15, 2026
Root cause found, and it is mine. Fix in #4280.
The hook's ServiceAccount does not exist by the time the pod runs. The e2e snapshot for the failing run holds
kafka-test-pre-delete-dd2zqwithserviceAccountName: kafka-test-pre-delete, while the namespace contains no ServiceAccount, Role or RoleBinding under that name. The Job is still there. A Job whose ServiceAccount is gone still starts a pod, and kubelet mounts the projected volume withca.crtandnamespacein it but no token, so client-go never assembles an in-cluster config and kubectl falls back to its default cluster. That is thelocalhost:8080in the log, and it is why this never looked like RBAC: the hook was not refused, it had nothing to authenticate with.The two policies were not the same:
helm.sh/hook-delete-policybefore-hook-creationbefore-hook-creation,hook-succeeded,hook-failedHelm reaps the credentials at the end of the run in either outcome and leaves the Job standing behind them. In
2b47c5a53the Job still carriedhook-succeededand the pair was consistent. I dropped it inceb9f211a, so the Job would outlive the run andttlSecondsAfterFinishedwould reclaim it, which is what keeps a readable record when the finalizer outlives the wait. I changed one half of the pair and never looked at the other.Confirmed by replay rather than by reading: the same digest-pinned image, with
KUBERNETES_SERVICE_HOSTset andca.crtpresent but the token file absent, prints the production message verbatim. Four other explanations were checked and are wrong: nothing in the tree setsautomountServiceAccountToken: false, the projected volume mountsdefaultMode: 420and is readable by uid 65534, the image carries noKUBECONFIG, and no Service namedkubernetesshadows the master one in that namespace.apps/kubernetesandapps/harborboth reach the apiserver from a hook and both keep the two policies symmetric, which is the shape #4280 restores. It also pins the policy on each credential object, because nothing compared them: the asymmetry rendered cleanly and passed 37 helm-unittest cases and 5 bats cases.One thing worth separating out, since it is not about this hook. The suite that caught this had never run on #3938. Its workflows sat at
action_requiredon all five heads the PR ever had, so the e2e lane first executed after the merge and reported twenty minutes later. Two rounds of review, mutation testing on both suites and a green unit lane did not substitute for it, and would not have: the manifest is correct in isolation and only a live uninstall shows the ordering.IvanHunters commentedon Sep 15, 2026
Correcting my previous comment: the delete-policy asymmetry was not the cause. Fixing it changed nothing, and the run that proved it also proved the diagnosis wrong. The ServiceAccount exists in that run's snapshot, created a second before the pod, and the hook fails exactly as before.
The actual cause is the
--request-timeout=20son the delete call. Any non-zero value for that flag makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that loads the ServiceAccount token never runs. kubectl then talks to its built-in default cluster,localhost:8080, is refused, and the script reads that refusal as a failed delete and exits 1.Measured on the pinned image, with the token and
ca.crtmounted andKUBERNETES_SERVICE_HOSTset, running the hook's own delete line:--request-timeout=20slocalhost:8080--request-timeout=1mlocalhost:8080--request-timeout=0The tree says the same thing independently. None of the hooks that work pass it:
apps/kubernetesnames it in a comment only,apps/harborandapps/mariadbnever mention it, andapps/tenant/templates/cleanup-job.yamlrecords that its DELETE calls "carry kubectl's default--request-timeoutof 0".Where my earlier comment went wrong is worth naming, because the shape of the mistake matters more than the mistake. I had measured that a missing token and a missing
KUBERNETES_SERVICE_HOSTproduce byte-identical output, so the log could not tell them apart. I then found a correlating detail in the snapshot, a ServiceAccount that was genuinely absent, and treated it as the cause. The second signal was not independent of the first; both only said "no in-cluster config", which is the one thing the log already said.apps/bucket/templates/hooks/cleanup-acme.yamlpasses--request-timeout=30son four calls against the same image and the same in-cluster ServiceAccount, so it has the same defect. No e2e suite covers that hook, which is why it is silent rather than fixed. Filed separately.#4280 now carries the real fix, and keeps the delete-policy change as what it turned out to be: a second, unrelated defect, where a Job outlives its own ServiceAccount.
fix(bucket): stop the ACME cleanup hook from talking to localhost, an…
fix(kafka): keep the pre-delete hook's credentials as long as its Job (…
[Backport release-1.6] fix(kafka): delete release topics before the t…
fix(kafka): delete the release's node pools on uninstall (#4490)