redis: the rfrm- Service (and <release>-external-lb) keeps no writable endpoint for the whole time the old master's node is down #4835

Summary

When the node running the Redis master loses power, Sentinel promotes the replica within seconds. The operator, however, does not move the redisfailovers-role=master pod label to the new master until the dead node is back. The rfrm-<release> Service, and with external: true the <release>-external-lb Service, both select on that label. So clients that use the plain Redis address instead of Sentinel cannot write for the whole node outage, even though the instance has a healthy, writable master.

Environment

• Cozystack v1.6.4, redis-operator ghcr.io/cozystack/cozystack/redis-operator:v1.3.0-rc1 (spotahome build). The relevant code is unchanged on main (freshworks-oss v3.3.5 plus Cozystack's patches), see "Where it comes from".
• Kubernetes v1.34.3 on Talos Linux, 3 nodes
• Provider: Hetzner Cloud (3 VMs)
• Redis app: replicas: 2, authEnabled: true, version: v8 (Redis 8.4.0), size: 1Gi on local storage; 3 Sentinels, one per node

Steps to reproduce

1. Create an HA Redis (replicas: 2) on a 3-node cluster, with the two Redis pods on different nodes.
2. Run two writers from a pod on another node, each doing RPUSH every 200 ms with a new connection and a 3 s timeout. One writes to rfrm-<release>:6379. The other first asks rfs-<release>:26379 for SENTINEL get-master-addr-by-name mymaster and writes to that address.
3. Hard power off the node running the master (no graceful shutdown). Keep it off for several minutes, then power it on.
4. Watch the redisfailovers-role labels on the Redis pods and the endpoints of rfrm-<release>.

Expected

After Sentinel promotes the replica, the operator moves redisfailovers-role=master on its next check, so rfrm-<release> follows the failover within roughly down-after-milliseconds + failover time + one resync period.

Actual

Power-cut test on 2026-10-09 (results-20261009-232355.txt; master's node powered off, back on later):

rfrm-<release> only recovered after the node was powered back on and the old pod had been handled. The node was Ready again 22 s after power-on, and Redis was back to 2 pods + 3 Sentinels with the replica attached after 158 s. For the whole time the node was down, the Service had no endpoint that accepted writes, while Sentinel and the promoted replica were healthy.

Where it comes from

The role labels are only set in CheckAllSlavesFromMaster (operator/redisfailover/service/check.go), which CheckAndHeal (operator/redisfailover/checker.go) reaches only after two gates:

1. IsRedisRunning → AreAllRunning returns false as soon as any Redis pod has a DeletionTimestamp (util.PodIsScheduling). After the taint-based eviction, the pod on the dead node stays Terminating until that node's kubelet confirms the deletion, because StatefulSet pods are not force-deleted. CheckAndHeal then returns early (Number of redis mismatch, waiting for redis statefulset reconcile) on every resync until the node is back.
2. Before the eviction, while the pod is still Running on the unreachable node: in spotahome v1.3.0-rc1 GetNumberMasters errors out on the unreachable IP, so the check stops before labelling. In freshworks v3.3.5 (main) GetNumberMasters and GetMasterIP skip unreachable pods, but CheckAllSlavesFromMaster still returns on the first GetSlaveOf error. It labels pods in list order, so when the dead old master comes first (for example rfr-<release>-0), it gets slave and the loop stops before the promoted pod gets master. We have reproduced this on v1.6.4 only; the main behaviour is from reading the code.

So no relabelling can happen while one Redis pod is on a dead node, which is exactly when the label matters.

Suggested direction

Relabelling does not depend on all replicas running; the source of truth is Sentinel's view of the master, or the reachable pods'. Options for the operator:

1. Before the IsRedisRunning early return, ask the Sentinels for the master (SENTINEL get-master-addr-by-name). If a quorum agrees and the address belongs to a Running, Ready, non-terminating Redis pod, label that pod master and every other pod slave, including terminating or unreachable ones.
2. Or, in CheckAllSlavesFromMaster, set all labels first and skip pods that are not Ready or are terminating when checking replication.

Either keeps the existing healing logic for the all-running case. For Cozystack: carry the operator fix as a patch under packages/system/redis-operator/images/redis-operator/patches/ until it is released upstream.

Workaround we use

A small HAProxy Deployment (2 replicas, anti-affinity, PDB) in front of the Redis pods, using HAProxy's documented Redis role check (option tcp-check, AUTH, INFO replication, expect string role:master). This way the plain address does not depend on the pod label. Same power-cut test with HAProxy (results-20261009-235851.txt): plain address outage 41 s, Sentinel-aware client 15 s, 0 of 1083 and 0 of 1080 acknowledged writes lost. It works, but it is an extra component that everyone who cannot use Sentinel-aware clients has to run. The tcp-balancer app cannot express it today, because it only allows fixed endpoints on ports 80/443 with a plain layer-4 check.