Repository navigation
Reconciling a tenant kubernetes app triggers MetalLB L2 VIP re-announcement and brief edge outage #2754
Description
Summary
Any Flux reconcile / Helm upgrade of a tenant kubernetes application (group apps.cozystack.io, kind Kubernetes) causes MetalLB to re-reconcile and re-elect the L2 owner of that tenant's LoadBalancer VIPs. The resulting ARP churn drops existing TCP connections, producing a brief edge outage (~minutes) for services exposed via those VIPs — even though no node, pod, or control plane is restarted and the rendered manifest barely changes.
Reproduction (observed on a production cluster)
A trivial change to a Kubernetes CR (bumping one node group's maxReplicas) was applied. The Helm manifest diff between the two release revisions was 2 lines (a MachineDeployment's maxSurge and the cluster-autoscaler max-size annotation) — no rollout occurred (machines/control-plane untouched).
Timeline (UTC):
T+0 Kubernetes CR patched (maxReplicas 3 -> 6)
10:04:40 Flux: HelmRelease UpgradeSucceeded
10:04:41 auth.<tenant> and vpn.<tenant> drop (connection refused)
10:04:47 MetalLB controller: burst of ServiceReconciler reconciles on the
tenant's *-ingress-* LoadBalancer services
10:05:15 speaker: serviceWithdrawn (reason: serviceDeleted)
10:07:15 speaker: serviceWithdrawn 10.x.x.90 (reason: notOwner) -> re-announce
10:07:29 speaker: serviceWithdrawn 10.x.x.100 (reason: notOwner) -> re-announce
... ~40 announce/withdraw events over ~5 min (baseline: ~0)
10:09:38 services recover
MetalLB speaker logs (L2) during the window show repeated withdrawing service announcement with reason: notOwner immediately followed by service has IP, announcing for the tenant ingress VIPs — i.e. the L2 owner keeps flapping between speaker nodes.
Reverting the change (another 2-line diff) reproduced the same churn, confirming it is the reconcile/apply itself, not the changed value.
Impact
- Brief but real outage of every service fronted by the affected L2 VIPs on each reconcile.
- Makes routine operations (scaling a node group, editing any tenant
kubernetesapp, and migrations that patch these CRs) unsafe outside a maintenance window.
Hypothesis
Flux re-applies the full release via 3-way merge, bumping resourceVersion/managedFields on ~all release objects → a wave of watch events → MetalLB re-runs L2 leader election for the tenant's VIPs. To confirm: identify why MetalLB re-reconciles VIPs that belong to other releases, and whether speaker memberlist / node events are involved.
Environment
- Cozystack tenant Kubernetes app (Kamaji + CAPI + KubeVirt workers)
- MetalLB in L2 (layer2) mode, edge ingress exposed via
LoadBalancer
Happy to provide more speaker/controller logs.
Activity
dosubot commentedon May 27, 2026
This is a well-documented issue. Based on the codebase, I can confirm the likely root cause and point to a few contributing factors:
Root Cause:
upgrade.force: trueon tenant addon HelmReleasesAll tenant Kubernetes addon HelmReleases (including ingress-nginx) are configured with
upgrade.force: true[1] and a 5-minute reconciliation interval [2]. Helm's--forceflag can cause Service objects to be replaced (delete + recreate) rather than patched in-place during upgrades. When a parentkubernetesCR changes and Flux propagates that into sub-HelmReleases, the LoadBalancer services created by the KubeVirt cloud provider in the infrastructure cluster get re-applied, triggering MetalLB to treat them as new services and re-run L2 leader election.Contributing factors:
helm.sh/resource-policyon Services — unlike CRDs and critical ConfigMaps which usehelm.sh/resource-policy: keepextensively across the platform [4], LoadBalancer services in tenant charts have no such protection.Potential mitigations (in rough order of impact):
upgrade.force: truefrom the ingress-nginx (and possibly other) addon HelmReleases — this is the most direct fix and would cause Helm to use a three-way strategic merge patch instead of replacement.metallb.universe.tf/loadBalancerIPsannotation to pin the VIP, reducing the chance MetalLB treats the re-applied Service as needing a fresh election.Related known MetalLB bug: Cozystack is on MetalLB v0.15.2 which also has a
ServiceL2Statusimmutability bug [7] — if your speaker nodes have experienced any reboot or failover recently, this could compound the flapping you're seeing, since the speaker can't properly update L2 status objects on a new leader node.Would you like me to look deeper into whether removing
upgrade.force: truefrom the ingress-nginx HelmRelease would be safe, or into the specific Helm template path that renders the tenant's LoadBalancer services?To reply, just mention @dosu.
Docs are dead. Just use Dosu.
lexfrei commentedon Jul 1, 2026
The root cause cited earlier in the thread (
upgrade.force: true) is no longer present inpackages/apps/kubernetes/templates/helmreleases/ingress-nginx.yamlon main — onlyouroboros.yamlstill usesforce. The separate Flux 3-way-merge / managedFields hypothesis cannot be confirmed from static source. Does this still reproduce on current main? Marking needs-information pending a runtime repro.github-actions commentedon Aug 31, 2026
This issue has had no activity for 60 days and was marked
lifecycle/stale.It will be closed in 14 days unless commented or labelled
lifecycle/frozen.github-actions commentedon Sep 14, 2026
Closed because no activity followed the
lifecycle/stalewarning.Reopen if the issue is still relevant.