platform: flux and cozystack-operator stay on a dead node (single replica, unbounded not-ready/unreachable tolerations) #4838

Summary

On a 3-node v1.6.4 cluster, several platform controllers run as a single replica, and two of them tolerate not-ready / unreachable forever. When the node they run on goes down, they stay bound to that node until it comes back. During that time no Package, HelmRelease or application change is reconciled. Customer traffic and database failover are not affected, but nothing new can be created or changed until the node returns.

Environment

• Cozystack v1.6.4, Talos v1.13.6, Kubernetes v1.34.3, 3 control-plane nodes (Hetzner Cloud)
• Pod eviction after 120 s for normal workloads (default-unreachable-toleration-seconds)

Observed

Steps to reproduce

1. Find the node running flux and cozystack-operator.
2. Hard power off that node.
3. Create or change any application, for example a Postgres instance, while the node is down.

Expected

The controllers move to a healthy node, or a standby replica takes over through leader election. The change is reconciled within a few minutes.

Actual

The pods stay Running on the dead node (Terminating never starts, because of the unbounded tolerations). The change is not reconciled until the node is powered back on.

Suggested direction

I understand the unbounded tolerations are needed during bootstrap, before the CNI is ready. Possible options:

1. Keep the bootstrap tolerations for node.cilium.io/agent-not-ready and uninitialized, but give not-ready / unreachable a bounded tolerationSeconds once the platform is installed.
2. Run 2 replicas with leader election where the controller supports it. Flux's controllers and controller-runtime based operators support --enable-leader-election.
3. Expose replicas / tolerations for linstor-csi-controller in the chart values.

We run our own staging and production on 3 equal nodes, so any one node loss should not stop the platform from reconciling.