Summary
On a 3-node v1.6.4 cluster, several platform controllers run as a single replica, and two of them tolerate not-ready / unreachable forever. When the node they run on goes down, they stay bound to that node until it comes back. During that time no Package, HelmRelease or application change is reconciled. Customer traffic and database failover are not affected, but nothing new can be created or changed until the node returns.
Environment
- Cozystack v1.6.4, Talos v1.13.6, Kubernetes v1.34.3, 3 control-plane nodes (Hetzner Cloud)
- Pod eviction after 120 s for normal workloads (
default-unreachable-toleration-seconds)
Observed
| Deployment |
Replicas |
Tolerations |
cozy-fluxcd/flux |
1 |
not-ready, unreachable, node.cilium.io/agent-not-ready, uninitialized, all without tolerationSeconds |
cozy-system/cozystack-operator |
1 |
same as above |
cozy-fluxcd/helm-controller-shard0 |
1 |
default (evicted after the cluster default) |
cozy-system/cozystack-controller |
1 |
default |
cozy-linstor/linstor-csi-controller |
1 |
only the DRBD taints; replicas and tolerations not exposed by the chart |
Steps to reproduce
- Find the node running
flux and cozystack-operator.
- Hard power off that node.
- Create or change any application, for example a Postgres instance, while the node is down.
Expected
The controllers move to a healthy node, or a standby replica takes over through leader election. The change is reconciled within a few minutes.
Actual
The pods stay Running on the dead node (Terminating never starts, because of the unbounded tolerations). The change is not reconciled until the node is powered back on.
Suggested direction
I understand the unbounded tolerations are needed during bootstrap, before the CNI is ready. Possible options:
- Keep the bootstrap tolerations for
node.cilium.io/agent-not-ready and uninitialized, but give not-ready / unreachable a bounded tolerationSeconds once the platform is installed.
- Run 2 replicas with leader election where the controller supports it. Flux's controllers and controller-runtime based operators support
--enable-leader-election.
- Expose
replicas / tolerations for linstor-csi-controller in the chart values.
We run our own staging and production on 3 equal nodes, so any one node loss should not stop the platform from reconciling.
Summary
On a 3-node v1.6.4 cluster, several platform controllers run as a single replica, and two of them tolerate
not-ready/unreachableforever. When the node they run on goes down, they stay bound to that node until it comes back. During that time no Package, HelmRelease or application change is reconciled. Customer traffic and database failover are not affected, but nothing new can be created or changed until the node returns.Environment
default-unreachable-toleration-seconds)Observed
cozy-fluxcd/fluxnot-ready,unreachable,node.cilium.io/agent-not-ready,uninitialized, all withouttolerationSecondscozy-system/cozystack-operatorcozy-fluxcd/helm-controller-shard0cozy-system/cozystack-controllercozy-linstor/linstor-csi-controllerSteps to reproduce
fluxandcozystack-operator.Expected
The controllers move to a healthy node, or a standby replica takes over through leader election. The change is reconciled within a few minutes.
Actual
The pods stay
Runningon the dead node (Terminatingnever starts, because of the unbounded tolerations). The change is not reconciled until the node is powered back on.Suggested direction
I understand the unbounded tolerations are needed during bootstrap, before the CNI is ready. Possible options:
node.cilium.io/agent-not-readyanduninitialized, but givenot-ready/unreachablea boundedtolerationSecondsonce the platform is installed.--enable-leader-election.replicas/tolerationsforlinstor-csi-controllerin the chart values.We run our own staging and production on 3 equal nodes, so any one node loss should not stop the platform from reconciling.