Skip to content

fix(seaweedfs): wait for ingress-nginx-system before creating Ingress - #2406

Merged
Andrei Kvapil (kvaps) merged 2 commits into
mainfrom
fix/seaweedfs-ingress-dependson
May 7, 2026
Merged

Andrei Kvapil (kvaps) merged 2 commits into
mainfrom
fix/seaweedfs-ingress-dependson

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Stacks on top of #2258. Adds dependsOn: ingress-nginx-system to the inner seaweedfs-system HelmRelease so flux blocks its install until the ingress chart is Ready.

Why

In #2258's E2E run (and the cozyreport-2258 bundle from this morning), seaweedfs-system failed to install with:

Helm install failed for release tenant-root/seaweedfs-system: 1 error occurred:
* Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io":
  failed to call webhook: Post "https://root-ingress-controller-admission.tenant-root.svc:443/networking/v1/ingresses?timeout=10s":
  dial tcp 10.96.33.42:443: connect: operation not permitted

Root cause class

The outer tenant HelmReleases (apps/tenant/templates/*.yaml) render one inner HR per subsystem and let flux reconcile them. Both seaweedfs and ingress outer HRs get applied in parallel, and each of them spawns its own inner HR — which flux then installs in parallel too.

When seaweedfs-system gets to the creating 39 resource(s) step, it tries to create the tenant Ingress. The apiserver calls the ingress-nginx ValidatingWebhookConfiguration (which uses failurePolicy: Fail by default). If the nginx controller pod/Service/endpoint state hasn't finished reconciling on the CNI yet, the connect to the ClusterIP fails with EPERM, the webhook errors, Helm aborts, and flux rolls back.

E2E recovers via the flux reconcile --force retry loop in hack/e2e-install-cozystack.bats, but it's flaky and adds minutes.

Fix

Introduce an explicit ordering dependency at the inner HR layer:

{{- if $ingress }}
dependsOn:
- name: ingress-nginx-system
{{- end }}

Flux waits for ingress-nginx-system to reach Ready=true before installing seaweedfs — which, under Flux's wait semantics, means the ingress-nginx Deployment is Available and its webhook endpoint is routable. The race disappears.

Guarded by {{- if $ingress }} so this only kicks in for tenants that actually have ingress enabled.

Test plan

  • Rerun E2E on Add vm-default-images package #2258 — seaweedfs-system should install on the first try without a Retry install iteration.
  • Verify rendered template still matches expectations when $ingress is unset (no dependsOn block).

Release note

[seaweedfs] Make tenant seaweedfs wait for ingress-nginx to be Ready before creating its Ingress resource, avoiding a webhook race that intermittently failed install on fresh clusters.

Summary by CodeRabbit

  • Bug Fixes
    • Improved deployment sequencing so the system now waits for the Ingress controller to be ready before creating or updating resources that include Ingress. This prevents premature deployments and related failures, reduces rollout errors, and provides more reliable, predictable deployments for applications relying on Ingress resources.

@coderabbitai

coderabbitai Bot commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack
No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1bfad425-90fa-4f2f-b83c-3e84d8e17ff5

📥 Commits

Reviewing files that changed from the base of the PR and between 7aadb4a and 026581e.

📒 Files selected for processing (1)
  • packages/extra/seaweedfs/templates/seaweedfs.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/extra/seaweedfs/templates/seaweedfs.yaml

📝 Walkthrough

Walkthrough

Modified the seaweedfs HelmRelease template to conditionally add spec.dependsOn referencing ingress-nginx-system when the tenant namespace equals the ingress namespace, causing Flux to wait for ingress readiness before reconciling.

Changes

HelmRelease dependency injection

Layer / File(s) Summary
Template conditional
packages/extra/seaweedfs/templates/seaweedfs.yaml
Adds a conditional HelmRelease.spec.dependsOn block that references ingress-nginx-system when $ingress == .Release.Namespace.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 A waiting song in YAML's flow,
"Hold here," the HelmRelease says slow,
Ingress ready, then we start,
Harmony kept, no tearing apart 🥕

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding a Flux dependency to wait for ingress-nginx-system before creating the seaweedfs Ingress, which directly matches the changeset's purpose.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/seaweedfs-ingress-dependson

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a race condition during the installation of SeaweedFS on fresh clusters. By introducing an explicit dependency on the ingress-nginx controller, it ensures that the ingress-nginx system is fully initialized and its admission webhooks are reachable before the SeaweedFS installation proceeds, thereby eliminating intermittent 'dial tcp' connection errors.

Highlights

  • Dependency Management: Added a conditional 'dependsOn' clause to the 'seaweedfs-system' HelmRelease to ensure it waits for 'ingress-nginx-system' to be ready.
  • Race Condition Resolution: Prevents intermittent installation failures caused by the apiserver attempting to call the ingress-nginx webhook before the service is fully routable.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@dosubot dosubot Bot added the size/XS This PR changes 0-9 lines, ignoring generated files label Apr 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a dependsOn field to the SeaweedFS template, ensuring that the ingress-nginx-system is ready before SeaweedFS resources are created when ingress is enabled. This change prevents intermittent failures during Ingress resource creation caused by the admission webhook not being fully reconciled. I have no feedback to provide.

Base automatically changed from feat/golden-images to main April 16, 2026 03:05
@myasnikovdaniil
myasnikovdaniil force-pushed the fix/seaweedfs-ingress-dependson branch from 6adae9b to e382968 Compare April 16, 2026 07:02

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct intent, but the guard condition creates a new regression for nested tenants. The {{- if $ingress }} gate fires whenever the tenant namespace has any ingress value — including values inherited from a parent tenant. For nested tenants that borrow their parent's ingress (a documented, supported topology — see packages/apps/tenant/README.md), $ingress is set to the parent's namespace name, but no ingress-nginx-system HelmRelease exists in the subtenant's namespace. Flux resolves dependsOn without an explicit namespace to the HelmRelease's own namespace, so the subtenant's seaweedfs-system gets stuck waiting on a dependency that cannot exist. One blocker inline. Two non-blocking notes on the same comment.

{{- if $ingress }}
# Ensure the ingress-nginx admission webhook is ready before creating Ingress resources.
dependsOn:
- name: ingress-nginx-system

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: this dependency breaks nested tenants that inherit ingress from a parent.

$ingress is computed at line 96 from .Values._namespace.ingress, which is populated by the tenant chart at packages/apps/tenant/templates/namespace.yaml:27-30:

{{- $ingress := $parentNamespace.ingress | default "" }}
{{- if .Values.ingress }}
{{- $ingress = $tenantName }}
{{- end }}

So _namespace.ingress is either this tenant's own namespace (when .Values.ingress: true) or the parent's namespace inherited through $parentNamespace.ingress. This is the documented nested-tenant model at packages/apps/tenant/README.md (the tenant-u1 / tenant-u2 example — u2 borrows u1's ingress).

For such a subtenant (ingress: false, seaweedfs: true, parent has ingress: true), $ingress is the parent namespace name (truthy), but ingress-nginx-system only gets rendered by packages/extra/ingress/templates/nginx-ingress.yaml — and the outer ingress HR that triggers it exists only when .Values.ingress: true at the tenant level (see packages/apps/tenant/templates/ingress.yaml:1). So in the subtenant's own namespace there is no ingress-nginx-system HR.

Flux dependsOn without a namespace: defaults to the HelmRelease's own namespace (meta.NamespacedObjectReference godoc; helm-controller checkDependencies). Missing dependencies don't time out — helm-controller requeues at DependencyRequeueInterval with DependencyNotReadyReason indefinitely. Combined with install.remediation.retries: -1 on this HR, the subtenant's seaweedfs-system will never install.

Current E2E at hack/e2e-install-cozystack.bats only covers tenant-root (self-ingress), so the regression will not show up in CI.

Suggested fix — gate on "this tenant owns its own ingress" so the dependency only appears when the target HR is actually local:

{{- if eq $ingress .Release.Namespace }}
# Ensure the ingress-nginx admission webhook is ready before creating Ingress resources.
dependsOn:
- name: ingress-nginx-system
  namespace: {{ .Release.Namespace }}
{{- end }}

This preserves the tenant-root bootstrap fix (where $ingress == "tenant-root" == .Release.Namespace) and skips the dependency for inherited cases (where the parent's ingress controller is already running anyway, so no race).

Two non-blocking notes while you're here:

  1. Every other dependsOn in the tree specifies namespace: explicitly (e.g., packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml:59). Worth matching that convention — it's what made the implicit-namespace behavior non-obvious in the first place.

  2. packages/extra/seaweedfs/templates/ingress.yaml creates a standalone seaweedfs-filer-external Ingress under .Values.filer.grpcHost via the outer HR, so it's not protected by this inner dependsOn. Probably out of scope for this PR (the E2E failure is specifically the inner HR), but a candidate for a follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 7aadb4a: gated the dependsOn on eq $ingress .Release.Namespace per your suggestion, and added explicit namespace: to match the rest of the tree.

Verified by rendering the template in three modes:

  • own ingress (tenant-root / _namespace.ingress = tenant-root) → dependsOn block emitted with namespace: tenant-root
  • inherited from parent (tenant-u2 / _namespace.ingress = tenant-parent) → dependsOn block skipped
  • no ingress (empty string) → dependsOn block skipped

Left the standalone seaweedfs-filer-external Ingress in templates/ingress.yaml untouched per your "out of scope" note — happy to do that as a follow-up.

@myasnikovdaniil
myasnikovdaniil marked this pull request as draft April 30, 2026 09:56
myasnikovdaniil added a commit that referenced this pull request May 7, 2026
…its ingress

Address review blocker on #2406: gate the dependsOn on
`eq $ingress .Release.Namespace` so sub-tenants inheriting ingress
from a parent namespace don't deadlock waiting for an HR that doesn't
exist locally. Also add explicit `namespace:` to match repo convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@github-actions github-actions Bot added the area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) label May 7, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review May 7, 2026 11:30
@dosubot dosubot Bot added area/extra Issues or PRs related to tenant-specific modules (packages/extra/) kind/bug Categorizes issue or PR as related to a bug labels May 7, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the SeaweedFS template to include a dependency on the ingress-nginx-system when the tenant namespace matches the ingress namespace, preventing deadlocks during Ingress resource creation. The review feedback indicates that the PR title and commit messages must be updated to follow the Conventional Commits format and include the mandatory Signed-off-by: trailer as per the repository style guide.

labels:
sharding.fluxcd.io/key: tenants
spec:
{{- if eq $ingress .Release.Namespace }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The PR title and commits do not follow the Conventional Commits format required by the repository style guide. Please update the PR title to follow the type(scope): description format (e.g., fix(seaweedfs): wait for ingress-nginx-system before creating Ingress). Also, ensure all commits have the Signed-off-by: trailer.

References
  1. Each commit must follow Conventional Commits format: type(scope): brief description. Each commit must have a Signed-off-by: trailer. (link)

Without this, the outer tenant HelmReleases apply both seaweedfs and
ingress in parallel, which in turn creates their inner HelmReleases
(seaweedfs-system and ingress-nginx-system) in parallel too. When
seaweedfs-system tries to create its Ingress before the nginx admission
webhook's Service/endpoint state has finished reconciling on the CNI,
the apiserver's webhook call fails with:

  failed calling webhook "validate.nginx.ingress.kubernetes.io":
  dial tcp ...:443: connect: operation not permitted

and flux rolls back the install. E2E recovers via the "Retry install"
loop but it's flaky and adds minutes to every run.

Adding dependsOn: ingress-nginx-system on the inner seaweedfs HR makes
flux block the install until the ingress chart is Ready, which means
the controller Deployment is Available and its webhook endpoint is
routable before the first Ingress is created.

Guarded by {{- if \$ingress }} so this only kicks in for tenants that
have ingress enabled (i.e. tenant-root in the default topology).

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
…its ingress

Address review blocker on #2406: gate the dependsOn on
`eq $ingress .Release.Namespace` so sub-tenants inheriting ingress
from a parent namespace don't deadlock waiting for an HR that doesn't
exist locally. Also add explicit `namespace:` to match repo convention.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil force-pushed the fix/seaweedfs-ingress-dependson branch from 7aadb4a to 026581e Compare May 7, 2026 11:38
@myasnikovdaniil myasnikovdaniil changed the title [seaweedfs] Wait for ingress-nginx-system before creating Ingress fix(seaweedfs): wait for ingress-nginx-system before creating Ingress May 7, 2026
@kvaps
Andrei Kvapil (kvaps) merged commit 99e7070 into main May 7, 2026
11 checks passed
@kvaps
Andrei Kvapil (kvaps) deleted the fix/seaweedfs-ingress-dependson branch May 7, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/extra Issues or PRs related to tenant-specific modules (packages/extra/) area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants