Repository navigation
fix(kafka): keep the pre-delete hook's credentials as long as its Job - #4280
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe Kafka pre-delete hook no longer passes a non-zero ChangesKafka pre-delete hook cleanup
Priority: ⬆️ High Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix · Severity of issue fixed: High Merge Risk: ⚪ Minimal · up to No concrete merge-blocking risk remains identified for the Kafka hook cleanup change. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The Job carries no hook-succeeded, so it outlives the hook run and ttlSecondsAfterFinished reclaims it, which is what leaves a readable record when the finalizer outlives the wait. Its ServiceAccount, Role and RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped them at the end of the run either way and left the Job behind them. A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the projected volume without a token, client-go finds no in-cluster config, kubectl falls back to localhost:8080, and the script takes its catch-all branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource goes, so the release never finishes deleting and helm-controller retries it: the e2e run on the merge commit shows six such pods in five minutes, in two namespaces, every one of them dead the same way. The hook meant to prevent one stuck Kafka release was instead stopping all of them. Give all four objects the same before-hook-creation policy, so the credentials live exactly as long as the consumer and the next run replaces them. Both other in-tree hooks that reach the apiserver, apps/kubernetes and apps/harbor, are symmetric this way already. The suite now pins the policy on each of the three, because nothing compared them: the asymmetry rendered cleanly, passed 37 helm-unittest cases and 5 bats cases, and only a cluster showed it. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
…hook A non-zero --request-timeout makes kubectl skip its in-cluster credentials altogether. client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl falls back to its built-in default cluster, localhost:8080, is refused, and the script reads the refusal as a failed delete and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource is removed, so no Kafka release could finish deleting and helm-controller retried forever. Measured on the pinned image rather than argued from the docs: with the token and ca.crt mounted and KUBERNETES_SERVICE_HOST set, the same delete reaches the cluster without the flag, goes to localhost:8080 with --request-timeout=20s, goes to localhost:8080 with 1m, and reaches the cluster again with 0. The three in-tree hooks that work do not pass it. apps/kubernetes names it in a comment only, apps/harbor and apps/mariadb never mention it, and apps/tenant records that its DELETE calls "carry kubectl's default --request-timeout of 0". The watch stays bounded by --timeout, and the Job by activeDeadlineSeconds, which is that same shape. The suite now rejects the flag. It needs the "=" because the explanation in the script names the flag too, and matching the bare name would fail on the comment rather than on a use. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
0eb03f0 to
e63b859
Compare
|
🤖 Completed: Fix CodeRabbit issues in PR #4280 — View commit |
…fferently The policies are right and nothing here changes them. What is missing is the reason, and this file has now been changed in both directions inside a week by people reasoning from the annotations alone, so the reason is the part worth keeping. The Job carries no hook-succeeded because the script exits 0 on the degraded path as well as the clean one, and its log is the only record that a finalizer outlived the wait; ttlSecondsAfterFinished reaps it on a timescale a human can act within. The ServiceAccount, Role and RoleBinding have no such clock, and before-hook-creation fires on a next activation that after the last uninstall never comes, so without a terminal policy they sit in the tenant namespace for the life of the cluster still granting delete over every KafkaTopic in it. Reaping them cannot strand a running pod, which is the objection the symmetrical shape was written against: helm applies hook-succeeded only once the hook is terminal, and activeDeadlineSeconds ends the Job inside the uninstall hook timeout. hook-failed would decide nothing, because a failing hook already reaps the earlier successful ones under hook-succeeded. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
Andrey Kolkov (androndo)
left a comment
There was a problem hiding this comment.
Verdict
LGTM
Re-reviewed at head 3a43bcf. The one commit since the earlier review is docs-only: it adds inline rationale and nothing else. The functional diff (dropping --request-timeout=20s, and dropping hook-failed from the RBAC hook-delete-policy) is unchanged and the unit tests pin both halves — revert either and the suite goes red, 37/37 pass otherwise. Nothing on the install/upgrade path is touched. One non-blocking MINOR below.
Findings
- [MINOR]
packages/apps/kafka/templates/hooks/delete.yaml:68, inline comments render verbatim into every customer Job manifest
Caveats
- Did not independently execute the
--request-timeoutin-cluster-credential mechanism (no live cluster; docker down). Not load-bearing: removing a per-request client timeout is non-harmful, the watch stays bounded by--timeout=180sand the Job byactiveDeadlineSeconds: 270, matching in-tree precedentpackages/apps/tenant/templates/cleanup-job.yaml. The described client-go behaviour (a non-default config override defeats the in-cluster fallback -> localhost:8080) is a real, known kubectl-in-pod gotcha. - RBAC lifecycle: the docs commit now records the intended design — the RBAC carries
hook-succeeded(unlike the ttl-reclaimed Job) so it does not outlive the release, and reaping is safe because helm reaps only once the Job is terminal, whichactiveDeadlineSecondsforces inside the hook timeout. On a genuinely failed uninstall the RBAC is kept alongside the retained Job and reclaimed bybefore-hook-creationon the next attempt. Round-1 naive's "orphan leak" MAJOR stays refuted. - Upgrade/fresh-install untouched: pre-delete hooks fire only on uninstall. RBAC stays namespaced with no scope widening.
Recommended follow-ups
packages/apps/bucket/templates/hooks/cleanup-acme.yamlpasses--request-timeout=30son in-cluster kubectl calls. If this PR's mechanism holds, those calls divert to localhost:8080 too; being fail-open (|| echo WARNING), the symptom is silently orphaned ACME resources rather than a wedged uninstall. Worth removing the flag there too (separate PR).
| - | | ||
| set -eu | ||
| echo "Deleting KafkaTopics for {{ .Release.Name }} before the topic-operator is removed" | ||
| # The call carries no request timeout, deliberately. Any non-zero |
There was a problem hiding this comment.
[MINOR] inline comments render verbatim into every customer Job manifest
Helm passes # comments straight through, so this 13-line client-go/#4276 explanation inside the Job's command[2] (plus the extra # rationale this commit adds on the ServiceAccount/Role/RoleBinding) lands in the rendered <release>-pre-delete manifest on every customer cluster, and goes stale the moment the code moves. It's a deliberate choice here — the commit exists to record the why — so keep it if you prefer. If you'd rather the manifest stayed lean, trim the Job block to a couple of lines plus an (#4276) ref and move the long form to the docs site. Non-blocking.
…d gate the class (#4289) ## What this PR does Fixes #4288, and closes the class behind it and #4276. `packages/apps/bucket/templates/hooks/cleanup-acme.yaml` passed `--request-timeout=30s` on all four of its kubectl calls. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka. Each call is guarded by `|| echo "WARNING: ..."` and the script sets no `-e`, so the hook exited 0 and printed `ACME cleanup complete.`, and `hook-succeeded` then deleted the Job holding those warnings. The ACME solver objects it exists to reap were left behind, with the only record removed moments after it was written. Its own suite matched the rendered command including the flag, so the broken form was pinned in place. The gate is the point of the change rather than a bonus. Two packages carried this, neither was visible in a rendered manifest, and both passed their unit suites. Nothing stops the next hook from reaching for the same flag, because it reads as ordinary defensive practice. `hack/check-hook-request-timeout.bats` fails any non-vendored template whose kubectl call carries a non-zero `--request-timeout`. The coarse part is stated in the file rather than left to be discovered: a template that names `KUBECONFIG` is skipped, because an explicit kubeconfig is not served by the in-cluster branch and the flag is harmless there. That is a per-file judgement, not per-command. It is what lets `apps/kubernetes` keep the flag on two calls that pass their own kubeconfig and stay green, and it is the first thing to tighten if a file ever mixes both styles. Checked in both directions rather than only on the clean tree: the gate passes as committed, fails if the flag is put back on any bucket call, fails if it is put back on the kafka call, and leaves the two `apps/kubernetes` files alone. It also carries a vacuity guard, because a collapsed glob would otherwise report success for having examined nothing. Bounding the work is still available where it belongs: `--timeout` for a watch, `activeDeadlineSeconds` for the Job, which this hook already had at 120s. **Depends on #4280** and is branched off it, so the gate is green here. #4280 removes the same flag from the kafka hook; merged before it, this check would be red on `main`. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. A hook's kubectl flags and a new `hack/*.bats` check; no values key, schema, API, default or naming contract moves, and nothing under `hack/` is moved or renamed, nor does any make target change what it does. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(bucket): the ACME cleanup hook now reaches the apiserver instead of reporting success while doing nothing ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved ACME cleanup reliability by preventing cleanup commands from targeting an unavailable local Kubernetes endpoint. * Improved Kafka uninstall cleanup reliability by ensuring the pre-delete hook retains access to required credentials throughout execution. * Preserved existing execution time limits and warning-based failure handling for cleanup operations. * **Tests** * Added coverage to prevent regressions in Kubernetes cleanup command configuration and hook resource lifecycle behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…opic operator is removed (#4456) Backport of #3938 and #4280 to `release-1.6`, together, because #3938 alone breaks every Kafka deletion (#4276) and #4280 is what fixes that. Kafka suite on this branch fails the delete step on 2 of the last 3 E2E runs on the container lane (#4328, #4377), both times with `UninstallFailed ... KafkaTopic/tenant-test/kafka-test-test-results termination timeout: context deadline exceeded`. Helm removes the topic operator together with the release, a KafkaTopic that still carries the `strimzi.io/topic-operator` finalizer then has nobody to release it, so uninstall waits out its timeout and leaves the topic behind, which is what #3793 hits on reinstall. When the operator wins the race the same delete takes 3s. The pre-delete hook from #3938 deletes the release topics while the operator is still running, #4280 makes it actually reach the apiserver and reaps its credentials. All ten commits cherry-picked clean with `-x`. Kafka on this branch is still ZooKeeper-based, the hook does not depend on that: it selects topics by `strimzi.io/cluster=<release>`, the label the 1.6 chart puts on them. ### Testing - `make unit-tests` green (kafka helm unit tests and `hack/kafka-pre-delete-hook.bats` included), POSIX sh sweep clean. - E2E on this PR runs the kafka suite, which now asserts both release topics are gone after deletion. ```release-note fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator ```
A hand backport that carries more than one change names all of them in one phrase -- "Backport of #3938 and #4280", or "Backport of #4253 to `release-1.6`, together with #3460" for a dependency pulled along -- and the audit read only the first number after "Backport of". The rest were linked to nothing. A labelled original in second place then read as MISSING while its backport PR was still open, instead of pending, and once that PR merged it counted as landed only if the branch history happened to prove it. Read the whole reference list the phrase carries, and the together-with form this repository writes. A list counts in full only when it visibly ends: at the end of the line or the sentence, or where a "to release-X.Y" clause names the target line, with the line name a whole word: release-1.6-fixes, release-1.6.1 and release-1.6.fixes are not lines, so punctuation after the name ends it only when a space or the end of the text follows. One that runs on into anything else -- "Backport of #10, #20 is not included", or "#10, #20 to follow in a separate PR" -- may be saying something about its later items, so only its first reference, the one the phrase names directly, is kept. The issue a backport fixes, or a CI run it cites further on in the body, is never taken for an original. A reference qualified with another repository is now skipped instead of read as a local number. That also tightens the old first-number rule, which accepted any owner/repo prefix: "Backport of other/repo#20" linked local #20, so a merged backport of something unrelated could turn that PR's MISSING verdict into backported. Only a bare #N or one qualified with the repository the backport PR itself lives in, compared without regard to case, is an original. Over every PR on release-1.4, release-1.5 and release-1.6 this changes the links of exactly four: #4456 and #4421 each gain #4280, #4377 gains #4231 and #4328 gains #3460. No verdict moves today, since none of the added originals carries a backport label. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
The audit started from labels and dropped every backport PR whose original was not a candidate for the branch. On release-1.6 that hid nine hand backports of unlabelled main PRs (#4431 to #4435, #4437, #4438, #4456 and #4475), and it had no way to notice two open backport PRs for the same originals: #4421 sat open next to #4456 for #3938 and #4280, and nothing reported it. Read the PRs on the branch from the side of the originals they claim as well, and report two more sections per branch. UNLABELLED lists each original claimed by backport PRs on the branch that is not a candidate for it, with every backport PR claiming it and its state, and says whether they backported it, only claim it with a PR still open, or were all closed. It never moves the exit code: the gate answers whether everything labelled landed, and an unlabelled backport can only add to a branch, never leave a labelled change off it. DUPLICATE lists each original claimed by two open backport PRs, or by an open one after another already merged, labelled or not. Both make the audit exit 1. One of the PRs is redundant, or the open one is the rest of a split backport; either way someone has to decide before the cut, and no verdict can say so, since a verdict settles on the first merged backport or reports the first open one as pending. A closed PR next to an open one is how a conflicting bot backport gets redone by hand and is not flagged. The titles of originals that no listing carries come from one GraphQL request for the whole run. A failed lookup costs the titles and nothing else: the URL is derived locally and the exit code is already settled. --json now emits an object per branch holding candidates, unlabelled and duplicates arrays, in place of the bare array of verdicts. On release-1.6 today this lists 13 unlabelled originals, the nine above among them, and three duplicates that are open right now: the bot's conflict drafts for #3936, #4254 and #4292 were left open next to their hand backports, and #4254 and #4292 each also have a fork PR open next to its reopening from a branch in this repository. #4421 is closed and shows up only as a closed claim on #4280. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
What this PR does
Fixes #4276. Since #3938 landed, no Kafka release finishes deleting: the pre-delete hook cannot reach the apiserver, exits non-zero, and a non-zero pre-delete hook returns from
Uninstall.Runbefore a single resource is removed. It is the inverse of what the hook was written for. #3793 was one release that would not delete; this made every release refuse to.The cause is
--request-timeout=20son the delete call. Any non-zero value for that flag makes kubectl skip its in-cluster credentials altogether: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl talks to its built-in default cluster,localhost:8080, is refused, and the script reads the refusal as a failed delete.Measured on the pinned image rather than argued from documentation. With the token and
ca.crtmounted andKUBERNETES_SERVICE_HOSTset, the same delete call reaches the cluster with no flag, goes tolocalhost:8080with--request-timeout=20s, goes tolocalhost:8080with1m, and reaches the cluster again with0.The tree agrees. None of the three hooks that work pass it:
apps/kubernetesnames it in a comment only,apps/harborandapps/mariadbnever mention it, andapps/tenant/templates/cleanup-job.yamlrecords that its DELETE calls "carry kubectl's default--request-timeoutof 0". The watch is still bounded by--timeout, and the Job byactiveDeadlineSeconds, which is that same shape.The second commit was a separate change found on the way, and review turned it around. It matched the credentials to the Job on
before-hook-creationalone, and a ServiceAccount, Role or RoleBinding has nothing likettlSecondsAfterFinishedto fall back on: helm removes a hook resource only when a policy says so, andbefore-hook-creationfires on a next activation that after the last uninstall never comes. All three would have sat in the tenant namespace for the life of the cluster, still granting delete over every KafkaTopic in it. They now carrybefore-hook-creation,hook-succeeded, the shapeapps/clickhouseandapps/kubernetes-nodesalready use, while the Job keepsbefore-hook-creationalone because the script exits 0 on the degraded path too and the Job's log is the only record that a finalizer outlived the wait. Reaping the credentials cannot strand a running pod: helm applieshook-succeededonly once the hook is terminal, andactiveDeadlineSeconds: 270ends the Job inside the 5m uninstall hook timeout. Neither of these caused #4276, and a cluster run confirmed as much.Both are pinned. Restoring the flag turns the suite red, with
0as well as with20s; the assertion matches--request-timeout=rather than the bare name, because the script's own comment names the flag. Dropping the reaping policy from the ServiceAccount alone, or from the Role alone, turns it red too, so the three are three assertions rather than one.Worth stating plainly, because it is the reason this reached
main: none of this is visible in a rendered manifest. The asymmetry and the flag both rendered cleanly, passed 37 helm-unittest cases and 5 bats cases through two rounds of review and per-branch mutation testing, and only a live uninstall showed either. The e2e lane never ran on #3938 while it was open, because its workflows sat ataction_requiredon all five heads a fork PR had.Screenshots
Not a UI change.
Downstream repositories
Walked the trigger map against the diff. This changes hook annotations and test assertions inside
packages/apps/kafka; no values key, schema, API, default or naming contract moves, and the app itself is neither added nor renamed.Release note
Summary by CodeRabbit
Bug Fixes
Tests