Repository navigation
kafka: the pre-delete hook cannot reach the apiserver, so every Kafka release now refuses to uninstall #4276
Description
Activity
Root cause found, and it is mine. Fix in #4280.
The hook's ServiceAccount does not exist by the time the pod runs. The e2e snapshot for the failing run holds
kafka-test-pre-delete-dd2zqwithserviceAccountName: kafka-test-pre-delete, while the namespace contains no ServiceAccount, Role or RoleBinding under that name. The Job is still there. A Job whose ServiceAccount is gone still starts a pod, and kubelet mounts the projected volume withca.crtandnamespacein it but no token, so client-go never assembles an in-cluster config and kubectl falls back to its default cluster. That is thelocalhost:8080in the log, and it is why this never looked like RBAC: the hook was not refused, it had nothing to authenticate with.The two policies were not the same:
object helm.sh/hook-delete-policyJob before-hook-creationServiceAccount, Role, RoleBinding before-hook-creation,hook-succeeded,hook-failedHelm reaps the credentials at the end of the run in either outcome and leaves the Job standing behind them. In
2b47c5a53the Job still carriedhook-succeededand the pair was consistent. I dropped it inceb9f211a, so the Job would outlive the run andttlSecondsAfterFinishedwould reclaim it, which is what keeps a readable record when the finalizer outlives the wait. I changed one half of the pair and never looked at the other.Confirmed by replay rather than by reading: the same digest-pinned image, with
KUBERNETES_SERVICE_HOSTset andca.crtpresent but the token file absent, prints the production message verbatim. Four other explanations were checked and are wrong: nothing in the tree setsautomountServiceAccountToken: false, the projected volume mountsdefaultMode: 420and is readable by uid 65534, the image carries noKUBECONFIG, and no Service namedkubernetesshadows the master one in that namespace.apps/kubernetesandapps/harborboth reach the apiserver from a hook and both keep the two policies symmetric, which is the shape #4280 restores. It also pins the policy on each credential object, because nothing compared them: the asymmetry rendered cleanly and passed 37 helm-unittest cases and 5 bats cases.One thing worth separating out, since it is not about this hook. The suite that caught this had never run on #3938. Its workflows sat at
action_requiredon all five heads the PR ever had, so the e2e lane first executed after the merge and reported twenty minutes later. Two rounds of review, mutation testing on both suites and a green unit lane did not substitute for it, and would not have: the manifest is correct in isolation and only a live uninstall shows the ordering.Correcting my previous comment: the delete-policy asymmetry was not the cause. Fixing it changed nothing, and the run that proved it also proved the diagnosis wrong. The ServiceAccount exists in that run's snapshot, created a second before the pod, and the hook fails exactly as before.
The actual cause is the
--request-timeout=20son the delete call. Any non-zero value for that flag makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that loads the ServiceAccount token never runs. kubectl then talks to its built-in default cluster,localhost:8080, is refused, and the script reads that refusal as a failed delete and exits 1.Measured on the pinned image, with the token and
ca.crtmounted andKUBERNETES_SERVICE_HOSTset, running the hook's own delete line:flag where kubectl went none in-cluster --request-timeout=20slocalhost:8080--request-timeout=1mlocalhost:8080--request-timeout=0in-cluster The tree says the same thing independently. None of the hooks that work pass it:
apps/kubernetesnames it in a comment only,apps/harborandapps/mariadbnever mention it, andapps/tenant/templates/cleanup-job.yamlrecords that its DELETE calls "carry kubectl's default--request-timeoutof 0".Where my earlier comment went wrong is worth naming, because the shape of the mistake matters more than the mistake. I had measured that a missing token and a missing
KUBERNETES_SERVICE_HOSTproduce byte-identical output, so the log could not tell them apart. I then found a correlating detail in the snapshot, a ServiceAccount that was genuinely absent, and treated it as the cause. The second signal was not independent of the first; both only said "no in-cluster config", which is the one thing the log already said.apps/bucket/templates/hooks/cleanup-acme.yamlpasses--request-timeout=30son four calls against the same image and the same in-cluster ServiceAccount, so it has the same defect. No e2e suite covers that hook, which is why it is silent rather than fixed. Filed separately.#4280 now carries the real fix, and keeps the delete-policy change as what it turned out to be: a second, unrelated defect, where a Job outlives its own ServiceAccount.
- added a commit that references this issue
on Sep 17, 2026 - added a commit that references this issue
on Sep 17, 2026 - added a commit that references this issue
on Sep 25, 2026 - added a commit that references this issue
on Sep 26, 2026
Describe the bug
The Kafka pre-delete hook added in #3938 cannot reach the apiserver.
kubectlinside it falls back tohttp://localhost:8080, every call is refused, the script takes its catch-all branch and exits 1, and a non-zero pre-delete hook returns fromUninstall.Runbefore any release resource is removed. helm-controller then retries the uninstall forever: the e2e run below shows sixkafka-test-pre-delete-*pods in five minutes, each one dying the same way.The net effect is the opposite of what the hook was written for. #3793 is a Kafka release that will not finish deleting; this makes every Kafka release refuse to delete, whether or not a topic ever held the finalizer.
Environment
main, ata19fd0274(the merge of fix(kafka): delete release topics before the topic operator is removed #3938)E2E (fork)run 34963349053To Reproduce
Install a Kafka application and delete it. The hook runs before anything is removed, fails, and the release stays.
Expected behaviour
The hook lists and deletes the release's KafkaTopics through the in-cluster credentials of the ServiceAccount it declares, and the uninstall proceeds.
Actual behaviour
Two suites go red as a result:
The first fails at
verify-pvc-reclaim-on-delete | DELETEonapps.cozystack.io/v1alpha1/Kafka @ tenant-test/test, which is the uninstall the hook is blocking.Additional context
localhost:8080is whatkubectluses when it found neither a kubeconfig nor in-cluster credentials, so this is not RBAC. Insufficient RBAC answers 403 and would have named the verb; the hook never got as far as a request to the cluster.What the root cause is not, as far as the tree shows. I compared the hook against the two that work:
packages/apps/kubernetes/templates/delete.yaml, which is alsopre-delete, andpackages/apps/harbor/templates/hooks/cleanup.yaml, which ispost-delete. All three carry the sameclastix/kubectl:v1.32image, the samerunAsUser: 65534withrunAsNonRootandreadOnlyRootFilesystem, the sameHOME=/tmp, the samepolicy.cozystack.io/allow-to-apiserver: "true"pod label, the sameserviceAccountNamenaming, and the same hook weights, 0 for the ServiceAccount, 5 for the Role and RoleBinding, 10 for the Job. Nothing in the tree setsautomountServiceAccountToken: falseon any of it. Whatever separates them is not visible in the manifests, so it needs a cluster to find.Worth recording about how this reached
main: the e2e lane never ran on #3938 while it was open. Its workflows sat ataction_requiredfor the whole life of the PR because it came from a fork, so the only checks that ever reported were the labelers and DCO. The suite ran for the first time after a maintainer authorised it, and it reported failure at 12:21, twenty minutes after the merge at 12:00.Checklist