Skip to content

[Backport release-1.6] fix(opensearch-operator): name the component reconciler a stall sits in - #4377

Merged
myasnikovdaniil merged 2 commits into
release-1.6from
backport-4333-to-release-1.6
Sep 25, 2026
Merged

myasnikovdaniil merged 2 commits into
release-1.6from
backport-4333-to-release-1.6

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Description

Backport of #4333 to release-1.6, together with #4231 which it depends on. 1.6 has no in-tree opensearch-operator build, so bot draft added only the Dockerfile and patch from #4333 would never reach an image. #4231 brings the build (image target, values wiring, root Makefile entry) plus its own single-replica fix. In root Makefile I kept only opensearch-operator line, redis-operator is not built in-tree on 1.6.

I built image locally, both patches apply to operator v2.8.0 and stall watchdog specs from #4333 pass.

@github-actions

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-4333-to-release-1.6
git worktree add --checkout .worktree/backport-4333-to-release-1.6 backport-4333-to-release-1.6
cd .worktree/backport-4333-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x 6a34c9f03717eac483904eaf86d6f337c6a73b47
git push --force-with-lease

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 00fadcb9-9b01-44aa-a204-3a0282710338

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@myasnikovdaniil
myasnikovdaniil force-pushed the backport-4333-to-release-1.6 branch from eb57599 to c617a9a Compare September 23, 2026 11:51
@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files labels Sep 23, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review September 23, 2026 12:03
…er bootstrap

The operator deletes the bootstrap pod as soon as initialization completes and
never looks at the voting configuration first. With one cluster_manager replica
there are exactly two eligible nodes, OpenSearch keeps the configuration at an
odd size of one, and the voter it keeps is the bootstrap node that formed the
cluster. Deleting that pod takes away the only vote there was, and the survivor
can never hold an election again, because changing the configuration itself
needs a quorum. The cluster is unusable about a minute after it installs, and a
node with no cluster-manager still answers the readiness probe, so nothing says
so until something restarts the pod.

Build the operator in-tree, the way kamaji and redis-operator already are, and
carry a patch that excludes the bootstrap node from the voting configuration and
waits for that to commit before deleting the pod. Reaching the cluster is a
precondition rather than a nicety: every failure keeps the pod and requeues,
because a bootstrap pod that outlives its purpose costs a pod, while a cluster
that cannot elect does not recover without a human.

Deleting a pod is a request rather than a wait, so clearing the exclusion
afterwards can find the node still in the cluster and fail. That path requeues
and is retried once the pod is really gone, which keeps a stale entry out of the
capped exclusion list; clearing stays at the API default wait_for_removal=true
so it cannot re-admit a node some other operation is still draining.

Upstream carries the same defect, tracked as
opensearch-project/opensearch-k8s-operator#1448.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit b222cb6)
Signed-off-by: Myasnikov Daniil <[email protected]>
The cluster controller runs nine component reconcilers in sequence and none
of them logs unconditionally when it is entered: the TLS one's only line
above its work sits on the no-security early return, which the chart in this
repository never renders, and the rolling restart one opens at V(1), off at
the verbosity the operator ships with. So a call that stops returning leaves
nothing behind at all. The operator log ends on whatever ran before it, the
object is silently no longer reconciled, and from the outside that is
indistinguishable from an operator with nothing to do. Two e2e runs ended
exactly that way, the operator going quiet at the moment the chart rewrote
the CR and never coming back: #4332.

Each component is now wrapped so a call still in flight after a minute
reports its own name and how long it has been running, and keeps reporting on
that interval while it is stuck. A single line dated at the start of a stall
does not answer the question a reader has five minutes in, which is whether
the call is still there. Nothing is written while a component behaves, so a
healthy cluster costs no extra lines.

The finalizer path is left alone: every DeleteResources at this version
returns immediately, so a watchdog there could only spawn goroutines for
calls that cannot block.

A diagnostic rather than a fix: drop it once the stall itself is understood.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 6a34c9f)
Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-4333-to-release-1.6 branch from c617a9a to ee8923f Compare September 24, 2026 14:50
myasnikovdaniil added a commit that referenced this pull request Sep 25, 2026
…opic operator is removed (#4456)

Backport of #3938 and #4280 to `release-1.6`, together, because #3938
alone breaks every Kafka deletion (#4276) and #4280 is what fixes that.

Kafka suite on this branch fails the delete step on 2 of the last 3 E2E
runs on the container lane (#4328, #4377), both times with
`UninstallFailed ... KafkaTopic/tenant-test/kafka-test-test-results
termination timeout: context deadline exceeded`. Helm removes the topic
operator together with the release, a KafkaTopic that still carries the
`strimzi.io/topic-operator` finalizer then has nobody to release it, so
uninstall waits out its timeout and leaves the topic behind, which is
what #3793 hits on reinstall. When the operator wins the race the same
delete takes 3s. The pre-delete hook from #3938 deletes the release
topics while the operator is still running, #4280 makes it actually
reach the apiserver and reaps its credentials.

All ten commits cherry-picked clean with `-x`. Kafka on this branch is
still ZooKeeper-based, the hook does not depend on that: it selects
topics by `strimzi.io/cluster=<release>`, the label the 1.6 chart puts
on them.

### Testing

- `make unit-tests` green (kafka helm unit tests and
`hack/kafka-pre-delete-hook.bats` included), POSIX sh sweep clean.
- E2E on this PR runs the kafka suite, which now asserts both release
topics are gone after deletion.

```release-note
fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator
```
@myasnikovdaniil
myasnikovdaniil merged commit e01d3f7 into release-1.6 Sep 25, 2026
41 of 42 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-4333-to-release-1.6 branch September 25, 2026 04:34
myasnikovdaniil added a commit that referenced this pull request Sep 25, 2026
A hand backport that carries more than one change names all of them in
one phrase -- "Backport of #3938 and #4280", or "Backport of #4253 to
`release-1.6`, together with #3460" for a dependency pulled along --
and the audit read only the first number after "Backport of". The rest
were linked to nothing. A labelled original in second place then read
as MISSING while its backport PR was still open, instead of pending,
and once that PR merged it counted as landed only if the branch history
happened to prove it.

Read the whole reference list the phrase carries, and the together-with
form this repository writes. A list counts in full only when it visibly
ends: at the end of the line or the sentence, or where a "to
release-X.Y" clause names the target line, with the line name a whole
word: release-1.6-fixes, release-1.6.1 and release-1.6.fixes are not
lines, so punctuation after the name ends it only when a space or the
end of the text follows. One that runs on into anything else --
"Backport of #10, #20 is not included", or "#10, #20 to follow in a
separate PR" -- may be saying something about its later items, so only
its first reference, the one the phrase names directly, is kept. The issue a backport fixes, or a CI run it cites further on in
the body, is never taken for an original.

A reference qualified with another repository is now skipped instead of
read as a local number. That also tightens the old first-number rule,
which accepted any owner/repo prefix: "Backport of other/repo#20" linked
local #20, so a merged backport of something unrelated could turn that
PR's MISSING verdict into backported. Only a bare #N or one qualified
with the repository the backport PR itself lives in, compared without
regard to case, is an original.

Over every PR on release-1.4, release-1.5 and release-1.6 this changes
the links of exactly four: #4456 and #4421 each gain #4280, #4377 gains
#4231 and #4328 gains #3460. No verdict moves today, since none of the
added originals carries a backport label.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants