Repository navigation
[Backport release-1.6] fix(opensearch-operator): name the component reconciler a stall sits in - #4377
Merged
Merged
Conversation
Author
|
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-4333-to-release-1.6
git worktree add --checkout .worktree/backport-4333-to-release-1.6 backport-4333-to-release-1.6
cd .worktree/backport-4333-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x 6a34c9f03717eac483904eaf86d6f337c6a73b47
git push --force-with-lease |
1 of 12 tasks
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: cozystack/cozystack/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
myasnikovdaniil
force-pushed
the
backport-4333-to-release-1.6
branch
from
September 23, 2026 11:51
eb57599 to
c617a9a
Compare
myasnikovdaniil
marked this pull request as ready for review
September 23, 2026 12:03
…er bootstrap The operator deletes the bootstrap pod as soon as initialization completes and never looks at the voting configuration first. With one cluster_manager replica there are exactly two eligible nodes, OpenSearch keeps the configuration at an odd size of one, and the voter it keeps is the bootstrap node that formed the cluster. Deleting that pod takes away the only vote there was, and the survivor can never hold an election again, because changing the configuration itself needs a quorum. The cluster is unusable about a minute after it installs, and a node with no cluster-manager still answers the readiness probe, so nothing says so until something restarts the pod. Build the operator in-tree, the way kamaji and redis-operator already are, and carry a patch that excludes the bootstrap node from the voting configuration and waits for that to commit before deleting the pod. Reaching the cluster is a precondition rather than a nicety: every failure keeps the pod and requeues, because a bootstrap pod that outlives its purpose costs a pod, while a cluster that cannot elect does not recover without a human. Deleting a pod is a request rather than a wait, so clearing the exclusion afterwards can find the node still in the cluster and fail. That path requeues and is retried once the pod is really gone, which keeps a stale entry out of the capped exclusion list; clearing stays at the API default wait_for_removal=true so it cannot re-admit a node some other operation is still draining. Upstream carries the same defect, tracked as opensearch-project/opensearch-k8s-operator#1448. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]> (cherry picked from commit b222cb6) Signed-off-by: Myasnikov Daniil <[email protected]>
The cluster controller runs nine component reconcilers in sequence and none of them logs unconditionally when it is entered: the TLS one's only line above its work sits on the no-security early return, which the chart in this repository never renders, and the rolling restart one opens at V(1), off at the verbosity the operator ships with. So a call that stops returning leaves nothing behind at all. The operator log ends on whatever ran before it, the object is silently no longer reconciled, and from the outside that is indistinguishable from an operator with nothing to do. Two e2e runs ended exactly that way, the operator going quiet at the moment the chart rewrote the CR and never coming back: #4332. Each component is now wrapped so a call still in flight after a minute reports its own name and how long it has been running, and keeps reporting on that interval while it is stuck. A single line dated at the start of a stall does not answer the question a reader has five minutes in, which is whether the call is still there. Nothing is written while a component behaves, so a healthy cluster costs no extra lines. The finalizer path is left alone: every DeleteResources at this version returns immediately, so a watchdog there could only spawn goroutines for calls that cannot block. A diagnostic rather than a fix: drop it once the stall itself is understood. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]> (cherry picked from commit 6a34c9f) Signed-off-by: Myasnikov Daniil <[email protected]>
myasnikovdaniil
force-pushed
the
backport-4333-to-release-1.6
branch
from
September 24, 2026 14:50
c617a9a to
ee8923f
Compare
myasnikovdaniil
added a commit
that referenced
this pull request
Sep 25, 2026
…opic operator is removed (#4456) Backport of #3938 and #4280 to `release-1.6`, together, because #3938 alone breaks every Kafka deletion (#4276) and #4280 is what fixes that. Kafka suite on this branch fails the delete step on 2 of the last 3 E2E runs on the container lane (#4328, #4377), both times with `UninstallFailed ... KafkaTopic/tenant-test/kafka-test-test-results termination timeout: context deadline exceeded`. Helm removes the topic operator together with the release, a KafkaTopic that still carries the `strimzi.io/topic-operator` finalizer then has nobody to release it, so uninstall waits out its timeout and leaves the topic behind, which is what #3793 hits on reinstall. When the operator wins the race the same delete takes 3s. The pre-delete hook from #3938 deletes the release topics while the operator is still running, #4280 makes it actually reach the apiserver and reaps its credentials. All ten commits cherry-picked clean with `-x`. Kafka on this branch is still ZooKeeper-based, the hook does not depend on that: it selects topics by `strimzi.io/cluster=<release>`, the label the 1.6 chart puts on them. ### Testing - `make unit-tests` green (kafka helm unit tests and `hack/kafka-pre-delete-hook.bats` included), POSIX sh sweep clean. - E2E on this PR runs the kafka suite, which now asserts both release topics are gone after deletion. ```release-note fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator ```
myasnikovdaniil
added a commit
that referenced
this pull request
Sep 25, 2026
A hand backport that carries more than one change names all of them in one phrase -- "Backport of #3938 and #4280", or "Backport of #4253 to `release-1.6`, together with #3460" for a dependency pulled along -- and the audit read only the first number after "Backport of". The rest were linked to nothing. A labelled original in second place then read as MISSING while its backport PR was still open, instead of pending, and once that PR merged it counted as landed only if the branch history happened to prove it. Read the whole reference list the phrase carries, and the together-with form this repository writes. A list counts in full only when it visibly ends: at the end of the line or the sentence, or where a "to release-X.Y" clause names the target line, with the line name a whole word: release-1.6-fixes, release-1.6.1 and release-1.6.fixes are not lines, so punctuation after the name ends it only when a space or the end of the text follows. One that runs on into anything else -- "Backport of #10, #20 is not included", or "#10, #20 to follow in a separate PR" -- may be saying something about its later items, so only its first reference, the one the phrase names directly, is kept. The issue a backport fixes, or a CI run it cites further on in the body, is never taken for an original. A reference qualified with another repository is now skipped instead of read as a local number. That also tightens the old first-number rule, which accepted any owner/repo prefix: "Backport of other/repo#20" linked local #20, so a merged backport of something unrelated could turn that PR's MISSING verdict into backported. Only a bare #N or one qualified with the repository the backport PR itself lives in, compared without regard to case, is an original. Over every PR on release-1.4, release-1.5 and release-1.6 this changes the links of exactly four: #4456 and #4421 each gain #4280, #4377 gains #4231 and #4328 gains #3460. No verdict moves today, since none of the added originals carries a backport label. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #4333 to
release-1.6, together with #4231 which it depends on. 1.6 has no in-tree opensearch-operator build, so bot draft added only the Dockerfile and patch from #4333 would never reach an image. #4231 brings the build (imagetarget, values wiring, rootMakefileentry) plus its own single-replica fix. In rootMakefileI kept only opensearch-operator line, redis-operator is not built in-tree on 1.6.I built image locally, both patches apply to operator v2.8.0 and stall watchdog specs from #4333 pass.