Repository navigation
Troubleshoot Kube-OVN and remove failed nodes - #273
Conversation
|
Caution Review failedThe pull request is closed. WalkthroughThis update adds two new troubleshooting sections to the documentation. The first section addresses Kube-OVN DaemonSet pod crashes due to a corrupted OVN database, providing cleanup steps and a DaemonSet manifest. The second section details the process for removing a failed node from a Cozystack Kubernetes cluster, including manual cleanup steps and commands. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~7 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
✅ Deploy Preview for cozystack ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello Andrei Kvapil (@kvaps), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the operational documentation by adding a new guide for configuring SeaweedFS in a multi-datacenter environment and providing detailed troubleshooting steps for common Kubernetes issues. It introduces solutions for Kube-OVN network problems and a comprehensive procedure for gracefully removing failed nodes from a cluster, including storage and etcd cleanup.
Highlights
- New SeaweedFS Multi-DC Documentation: Added a new documentation page (
content/en/docs/operations/seaweedfs-multidc.md) detailing the configuration of SeaweedFS for multi-datacenter deployments, including YAML examples forTenantandSeaweedFScustom resources withMultiZoneandClienttopologies. - Kube-OVN Troubleshooting Guide: Introduced a new section in the
content/en/docs/operations/troubleshooting.mdguide (starting at line 133) providing steps to diagnose and resolve Kube-OVN crashes, specifically addressing corrupted OVN databases with a cleanup DaemonSet and associated commands. - Failed Kubernetes Node Removal: Added a comprehensive guide in
content/en/docs/operations/troubleshooting.md(starting at line 233) for safely removing failed Kubernetes nodes. This includes steps forkubectl delete node, etcd member cleanup for control-plane nodes, and manual cleanup of bound PVCs and pending pods usingkubectlandjq/awkcommands. - LINSTOR Resource Recovery: Included instructions within the failed node removal guide on using
linstor advise resourceto identify and resolve LINSTOR storage replica issues after node failures, along with commands to restore desired replica counts.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | Gemini (@gemini-code-assist) | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
325ba36 to
ec6bf1a
Compare
Andrei Kvapil (kvaps)
changed the title
There was a problem hiding this comment.
Code Review
This pull request adds documentation for troubleshooting a failed node and a new page for SeaweedFS multi-DC configuration. The SeaweedFS documentation has a minor typo in an example hostname. The troubleshooting guide has an incomplete sentence, inconsistent markdown heading levels, a potentially dangerous command that could delete unrelated pods, and a Kubernetes manifest for a cleanup DaemonSet that seems to have a logic issue in its scheduling rules.
1a8bc47 to
91ba948
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
content/en/docs/operations/troubleshooting.md (4)
138-144: Add a language specifier to the fenced block
markdownlintflags this block (# kubectl get pod …) because it lacks a language tag.
Usebashso syntax highlighting works and the linter passes.-``` +# ```bash
146-161: Missing language on the log excerpt blockSame MD040 violation as above. The block only shows raw log lines, so
textis sufficient.-``` +# ```text
241-241: Heading level jumps from H2 to H4Markdown guideline MD001: increment levels one at a time. Change
####to###(and subsequent step headings likewise).
245-248: Add language specifier to code blockThe
kubectl delete nodeexample should declarebash.-``` +# ```bash
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
content/en/docs/operations/troubleshooting.md(1 hunks)
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
content/en/docs/operations/troubleshooting.md
138-138: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
146-146: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
241-241: Heading levels should only increment by one level at a time
Expected: h3; Actual: h4
(MD001, heading-increment)
245-245: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
257-257: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (1)
content/en/docs/operations/troubleshooting.md (1)
277-284: Quoting & safety for generated delete commands
| sh -xwill execute whateverjqprints; if aclaimRefis missing.namespaceor.namethis will expand to an empty token and raise errors.
Consider guarding with-r select(.claimRef) …or removing the automatic pipe-to-shell and asking users to review first.
At minimum document the risk.
| containers: | ||
| - name: cleanup | ||
| image: busybox | ||
| command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/*; rm -rf /host-config-ovn/.*; exec sleep infinity"] |
There was a problem hiding this comment.
🛠️ Refactor suggestion
rm -rf /host-config-ovn/.* tries to delete . and ..
The glob .* expands to . and .., producing noisy “cannot remove” errors.
Use a safer pattern that excludes them:
-command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/*; rm -rf /host-config-ovn/.*; exec sleep infinity"]
+command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/* /host-config-ovn/.[!.]* /host-config-ovn/..?*; exec sleep infinity"]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/*; rm -rf /host-config-ovn/.*; exec sleep infinity"] | |
| command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/* /host-config-ovn/.[!.]* /host-config-ovn/..?*; exec sleep infinity"] |
🤖 Prompt for AI Agents
In content/en/docs/operations/troubleshooting.md at line 195, the command uses
the glob pattern 'rm -rf /host-config-ovn/.*' which attempts to delete '.' and
'..' directories, causing errors. Replace this pattern with a safer one that
excludes '.' and '..', such as using a pattern like '/host-config-ovn/.[!.]*' or
another method that avoids matching these special directories to prevent the
noisy errors.
| kubectl delete ds | ||
| kubectl get pod -n cozy-kubeovn | ||
| ``` |
There was a problem hiding this comment.
kubectl delete ds without arguments is hazardous
Running kubectl delete ds deletes all DaemonSets in the current namespace.
Specify the name and namespace explicitly, e.g.:
-kubectl delete ds
+kubectl delete ds ovn-cleanup -n cozy-kubeovn📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| kubectl delete ds | |
| kubectl get pod -n cozy-kubeovn | |
| ``` | |
| kubectl delete ds ovn-cleanup -n cozy-kubeovn | |
| kubectl get pod -n cozy-kubeovn |
🤖 Prompt for AI Agents
In content/en/docs/operations/troubleshooting.md around lines 226 to 228, the
command `kubectl delete ds` is used without specifying a DaemonSet name or
namespace, which can delete all DaemonSets in the current namespace
unintentionally. Update the example to include the DaemonSet name and namespace
explicitly, such as `kubectl delete ds <daemonset-name> -n <namespace>`, to
prevent accidental mass deletion.
| ``` | ||
|
|
||
| To resolve this issue, you can clean up the OVN database. This involves running a DaemonSet that removes the OVN configuration files from each node. | ||
| It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes A |
There was a problem hiding this comment.
Andrei Kvapil (@kvaps) "from the Kubernetes A..."?
Kubernetes API?
Signed-off-by: Andrei Kvapil <[email protected]>
Signed-off-by: Nick Volynkin <[email protected]>
57df6a7 to
d67060f
Compare
Nick Volynkin (NickVolynkin)
left a comment
Nick Volynkin (NickVolynkin)
left a comment
There was a problem hiding this comment.
LGTM
2 participants
fixes cozystack/cozystack#4
Summary by CodeRabbit