Repository navigation
fix(seaweedfs): wait for ingress-nginx-system before creating Ingress - #2406
Conversation
|
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughModified the seaweedfs HelmRelease template to conditionally add ChangesHelmRelease dependency injection
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a race condition during the installation of SeaweedFS on fresh clusters. By introducing an explicit dependency on the ingress-nginx controller, it ensures that the ingress-nginx system is fully initialized and its admission webhooks are reachable before the SeaweedFS installation proceeds, thereby eliminating intermittent 'dial tcp' connection errors. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request adds a dependsOn field to the SeaweedFS template, ensuring that the ingress-nginx-system is ready before SeaweedFS resources are created when ingress is enabled. This change prevents intermittent failures during Ingress resource creation caused by the admission webhook not being fully reconciled. I have no feedback to provide.
6adae9b to
e382968
Compare
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
Correct intent, but the guard condition creates a new regression for nested tenants. The {{- if $ingress }} gate fires whenever the tenant namespace has any ingress value — including values inherited from a parent tenant. For nested tenants that borrow their parent's ingress (a documented, supported topology — see packages/apps/tenant/README.md), $ingress is set to the parent's namespace name, but no ingress-nginx-system HelmRelease exists in the subtenant's namespace. Flux resolves dependsOn without an explicit namespace to the HelmRelease's own namespace, so the subtenant's seaweedfs-system gets stuck waiting on a dependency that cannot exist. One blocker inline. Two non-blocking notes on the same comment.
| {{- if $ingress }} | ||
| # Ensure the ingress-nginx admission webhook is ready before creating Ingress resources. | ||
| dependsOn: | ||
| - name: ingress-nginx-system |
There was a problem hiding this comment.
Blocker: this dependency breaks nested tenants that inherit ingress from a parent.
$ingress is computed at line 96 from .Values._namespace.ingress, which is populated by the tenant chart at packages/apps/tenant/templates/namespace.yaml:27-30:
{{- $ingress := $parentNamespace.ingress | default "" }}
{{- if .Values.ingress }}
{{- $ingress = $tenantName }}
{{- end }}
So _namespace.ingress is either this tenant's own namespace (when .Values.ingress: true) or the parent's namespace inherited through $parentNamespace.ingress. This is the documented nested-tenant model at packages/apps/tenant/README.md (the tenant-u1 / tenant-u2 example — u2 borrows u1's ingress).
For such a subtenant (ingress: false, seaweedfs: true, parent has ingress: true), $ingress is the parent namespace name (truthy), but ingress-nginx-system only gets rendered by packages/extra/ingress/templates/nginx-ingress.yaml — and the outer ingress HR that triggers it exists only when .Values.ingress: true at the tenant level (see packages/apps/tenant/templates/ingress.yaml:1). So in the subtenant's own namespace there is no ingress-nginx-system HR.
Flux dependsOn without a namespace: defaults to the HelmRelease's own namespace (meta.NamespacedObjectReference godoc; helm-controller checkDependencies). Missing dependencies don't time out — helm-controller requeues at DependencyRequeueInterval with DependencyNotReadyReason indefinitely. Combined with install.remediation.retries: -1 on this HR, the subtenant's seaweedfs-system will never install.
Current E2E at hack/e2e-install-cozystack.bats only covers tenant-root (self-ingress), so the regression will not show up in CI.
Suggested fix — gate on "this tenant owns its own ingress" so the dependency only appears when the target HR is actually local:
{{- if eq $ingress .Release.Namespace }}
# Ensure the ingress-nginx admission webhook is ready before creating Ingress resources.
dependsOn:
- name: ingress-nginx-system
namespace: {{ .Release.Namespace }}
{{- end }}This preserves the tenant-root bootstrap fix (where $ingress == "tenant-root" == .Release.Namespace) and skips the dependency for inherited cases (where the parent's ingress controller is already running anyway, so no race).
Two non-blocking notes while you're here:
-
Every other
dependsOnin the tree specifiesnamespace:explicitly (e.g.,packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml:59). Worth matching that convention — it's what made the implicit-namespace behavior non-obvious in the first place. -
packages/extra/seaweedfs/templates/ingress.yamlcreates a standaloneseaweedfs-filer-externalIngress under.Values.filer.grpcHostvia the outer HR, so it's not protected by this innerdependsOn. Probably out of scope for this PR (the E2E failure is specifically the inner HR), but a candidate for a follow-up.
There was a problem hiding this comment.
Addressed in 7aadb4a: gated the dependsOn on eq $ingress .Release.Namespace per your suggestion, and added explicit namespace: to match the rest of the tree.
Verified by rendering the template in three modes:
- own ingress (
tenant-root/_namespace.ingress = tenant-root) →dependsOnblock emitted withnamespace: tenant-root - inherited from parent (
tenant-u2/_namespace.ingress = tenant-parent) →dependsOnblock skipped - no ingress (empty string) →
dependsOnblock skipped
Left the standalone seaweedfs-filer-external Ingress in templates/ingress.yaml untouched per your "out of scope" note — happy to do that as a follow-up.
…its ingress Address review blocker on #2406: gate the dependsOn on `eq $ingress .Release.Namespace` so sub-tenants inheriting ingress from a parent namespace don't deadlock waiting for an HR that doesn't exist locally. Also add explicit `namespace:` to match repo convention. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
There was a problem hiding this comment.
Code Review
This pull request updates the SeaweedFS template to include a dependency on the ingress-nginx-system when the tenant namespace matches the ingress namespace, preventing deadlocks during Ingress resource creation. The review feedback indicates that the PR title and commit messages must be updated to follow the Conventional Commits format and include the mandatory Signed-off-by: trailer as per the repository style guide.
| labels: | ||
| sharding.fluxcd.io/key: tenants | ||
| spec: | ||
| {{- if eq $ingress .Release.Namespace }} |
There was a problem hiding this comment.
The PR title and commits do not follow the Conventional Commits format required by the repository style guide. Please update the PR title to follow the type(scope): description format (e.g., fix(seaweedfs): wait for ingress-nginx-system before creating Ingress). Also, ensure all commits have the Signed-off-by: trailer.
References
- Each commit must follow Conventional Commits format: type(scope): brief description. Each commit must have a Signed-off-by: trailer. (link)
Without this, the outer tenant HelmReleases apply both seaweedfs and
ingress in parallel, which in turn creates their inner HelmReleases
(seaweedfs-system and ingress-nginx-system) in parallel too. When
seaweedfs-system tries to create its Ingress before the nginx admission
webhook's Service/endpoint state has finished reconciling on the CNI,
the apiserver's webhook call fails with:
failed calling webhook "validate.nginx.ingress.kubernetes.io":
dial tcp ...:443: connect: operation not permitted
and flux rolls back the install. E2E recovers via the "Retry install"
loop but it's flaky and adds minutes to every run.
Adding dependsOn: ingress-nginx-system on the inner seaweedfs HR makes
flux block the install until the ingress chart is Ready, which means
the controller Deployment is Available and its webhook endpoint is
routable before the first Ingress is created.
Guarded by {{- if \$ingress }} so this only kicks in for tenants that
have ingress enabled (i.e. tenant-root in the default topology).
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
…its ingress Address review blocker on #2406: gate the dependsOn on `eq $ingress .Release.Namespace` so sub-tenants inheriting ingress from a parent namespace don't deadlock waiting for an HR that doesn't exist locally. Also add explicit `namespace:` to match repo convention. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
7aadb4a to
026581e
Compare
What this PR does
Stacks on top of #2258. Adds
dependsOn: ingress-nginx-systemto the innerseaweedfs-systemHelmRelease so flux blocks its install until the ingress chart is Ready.Why
In #2258's E2E run (and the
cozyreport-2258bundle from this morning),seaweedfs-systemfailed to install with:Root cause class
The outer tenant HelmReleases (
apps/tenant/templates/*.yaml) render one inner HR per subsystem and let flux reconcile them. Bothseaweedfsandingressouter HRs get applied in parallel, and each of them spawns its own inner HR — which flux then installs in parallel too.When
seaweedfs-systemgets to thecreating 39 resource(s)step, it tries to create the tenant Ingress. The apiserver calls the ingress-nginxValidatingWebhookConfiguration(which usesfailurePolicy: Failby default). If the nginx controller pod/Service/endpoint state hasn't finished reconciling on the CNI yet, the connect to the ClusterIP fails withEPERM, the webhook errors, Helm aborts, and flux rolls back.E2E recovers via the
flux reconcile --forceretry loop inhack/e2e-install-cozystack.bats, but it's flaky and adds minutes.Fix
Introduce an explicit ordering dependency at the inner HR layer:
Flux waits for
ingress-nginx-systemto reachReady=truebefore installing seaweedfs — which, under Flux's wait semantics, means the ingress-nginx Deployment is Available and its webhook endpoint is routable. The race disappears.Guarded by
{{- if $ingress }}so this only kicks in for tenants that actually have ingress enabled.Test plan
seaweedfs-systemshould install on the first try without aRetry installiteration.$ingressis unset (nodependsOnblock).Release note
Summary by CodeRabbit