Repository navigation
Client stuck in isOpen && !isReady forever after server restart — reconnect loop dies silently #3346
Description
Environment
redis (node-redis) |
6.1.0 |
| Node.js | 22 LTS (also seen on 24.x) |
| Redis server | 7.x (redis:7-alpine, --appendonly yes) |
| OS | reproduced on both Linux (CI containers) and Windows 11 |
| Topology | single client, single standalone server, no cluster, no sentinel |
Summary
After the server goes away abruptly and comes back (SHUTDOWN NOSAVE followed by a
restart), the client can land in a state where:
client.isOpen === trueclient.isReady === false- no further lifecycle events are ever emitted — no
error, noreconnecting,
noend, indefinitely reconnectStrategyis never called againawait client.connect()rejects withSocket already opened, so the application
cannot recover the client either
The reconnect loop appears to have terminated without emitting a terminal event, while
the socket-state flag still claims the client is open. From the application's point of
view the client is neither usable nor recoverable, and — critically — nothing tells
it so. A process that reports readiness from isOpen will report healthy forever.
It reproduces roughly 50% of the time in a bare loop with no application load.
Expected
One of:
- the reconnect loop keeps retrying per
reconnectStrategyuntil it succeeds (the
documented behaviour), or - the client gives up loudly — emits
error/endand flipsisOpentofalse, so
connect()can be called again.
Either is actionable. The current outcome — open, not ready, silent, and refusing
connect() — is not.
Actual
The client sits at isOpen: true, isReady: false with a dead retry chain and emits
nothing further. Observed in production-shaped code as "the pod never reconnects
after a Redis failover".
Minimal reproduction
Requires Docker (any Redis 7 works; the AOF flag only keeps the dataset around and is
not needed to trigger the bug).
// repro.js — node repro.js
import { execFileSync } from 'node:child_process';
import { createClient } from 'redis';
const NAME = 'node-redis-zombie-repro';
const PORT = 6399;
const sh = (...args) => execFileSync('docker', args, { stdio: 'ignore' });
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function attempt(n) {
try {
sh('rm', '-f', NAME);
} catch {
/* not running */
}
sh('run', '-d', '--name', NAME, '-p', `${PORT}:6379`, 'redis:7-alpine');
await sleep(1000);
const client = createClient({
socket: {
host: '127.0.0.1',
port: PORT,
connectTimeout: 2000,
reconnectStrategy: (retries) => Math.min(50 * 2 ** retries, 2000),
},
disableOfflineQueue: true,
});
const events = [];
for (const event of ['connect', 'ready', 'reconnecting', 'error', 'end']) {
client.on(event, () => events.push([event, Date.now()]));
}
await client.connect();
// Abrupt server death, then bring it back.
try {
sh('exec', NAME, 'redis-cli', 'SHUTDOWN', 'NOSAVE');
} catch {
/* SHUTDOWN kills the connection; a non-zero exit here is expected */
}
await sleep(500);
sh('start', NAME);
// A healthy reconnect loop emits at least every ~2s (the backoff cap).
// Watch for 15s of total silence while open-but-not-ready.
for (let i = 0; i < 30; i += 1) {
await sleep(500);
if (client.isReady) {
console.log(`attempt ${n}: recovered`);
await client.destroy();
sh('rm', '-f', NAME);
return false;
}
}
const quietMs = Date.now() - (events.at(-1)?.[1] ?? 0);
const zombie = client.isOpen && !client.isReady && quietMs > 10_000;
console.log(
`attempt ${n}: isOpen=${client.isOpen} isReady=${client.isReady} ` +
`quietMs=${quietMs} lastEvent=${events.at(-1)?.[0]} ZOMBIE=${zombie}`,
);
if (zombie) {
// The application has no way out from here:
await client.connect().catch((err) => console.log(` connect() -> ${err.message}`));
}
await client.destroy().catch(() => {});
sh('rm', '-f', NAME);
return zombie;
}
let zombies = 0;
for (let n = 1; n <= 10; n += 1) if (await attempt(n)) zombies += 1;
console.log(`\n${zombies}/10 attempts ended in a zombie client`);Typical output on an affected run:
attempt 1: recovered
attempt 2: isOpen=true isReady=false quietMs=14512 lastEvent=error ZOMBIE=true
connect() -> Socket already opened
attempt 3: isOpen=true isReady=false quietMs=14498 lastEvent=error ZOMBIE=true
connect() -> Socket already opened
attempt 4: recovered
...
5/10 attempts ended in a zombie client
Why this matters
isOpen is the natural signal for a Kubernetes readiness probe. In the zombie state it
stays true, so the pod keeps reporting healthy and keeps receiving traffic while every
command fails. There is no event to react to and no supported way to rebuild the client
in place.
Workaround we shipped
A watchdog that treats lifecycle-event silence as the symptom, since the flags lie:
a healthy retry loop emits at least every backoff-cap interval (~2 s), so
open-but-not-ready with no event for ≥ 10 s is structurally impossible for a live loop.
On detection it hard-recycles the connection with destroy() + connect(), which
revives the client immediately (6/6 in the repro loop above).
Happy to turn the watchdog into a PR if a reconnectStrategy-side fix is not
straightforward.
Activity
PavelPashov commentedon Jul 20, 2026
We couldn’t reproduce this: the supplied script recovered successfully in all 10 runs. The issue body also contains unremoved drafting instructions, internal references marked “do not paste,” and empty template fields; please clean it up and provide a reproducible case before we investigate further.
[-]Redis Zombie Reconnect[/-][+]Client stuck in isOpen && !isReady forever after server restart — reconnect loop dies silently[/+]vizyonok commentedon Jul 20, 2026
Apologies for the state of the original post — it went up with our internal drafting scaffolding pasted in by mistake. The body is now cleaned to just the report, and the title is fixed.
On reproduction: the race is timing-sensitive, so the hit rate varies a lot by machine. We see roughly 50% per attempt on two quite different setups (Windows 11 bare metal, and Linux CI-class runners); a fast machine with little contention may well recover every time. Two suggestions that should widen the window if your 10/10 recovered:
docker run --cpus=0.5 ...) or run the loop under parallel CPU load - the failure needs the restart to land at a particular point in the client's retry cycle, and contention makes that alignment more likely.One diagnostic detail from our zombie runs: the last lifecycle event emitted is an
error, then nothing -reconnectStrategyis never consulted again, andconnect()rejects withSocket already opened. If it would help, we're happy to run an instrumented build with extra socket-state logging on one of the machines where this reproduces and attach the trace from a zombie run - tell us what you'd like logged.nkaradzhov commentedon Jul 28, 2026
Thanks @vizyonok, we were able to reproduce this and confirm the root cause: if the socket dies while the connection initiator is suspended on async work (e.g. DNS resolution, async credentials provider), the reconnect loop stays suspended forever with no terminal event — exactly the
isOpen: true / isReady: falsestate you described. Your script uses127.0.0.1, which skips the DNS step, which is why it didn't reproduce for us as-is.Fix is up in #3374, including a deterministic regression test for the interleaving.
vizyonok commentedon Jul 28, 2026
Thanks for tracking that down — the async-suspension mechanism matches what we observed exactly. We'll test our workload against the fix once it's in a release.
nkaradzhov commentedon Jul 28, 2026
We are aiming to release a beta minor version Wednesday and then release again on Friday. This should catch the Wednesday release
fix(client): reject in-flight connect attempt when socket dies during…
vizyonok commentedon Aug 2, 2026
Confirming from our side: with stable 6.2.0 our repro loop from this issue runs 25/25 clean — every attempt recovers, no zombie state — and we've moved our production pin from 6.1.0 to 6.2.0.
In fairness, our 6.1.0 control run came back clean today too — the race is as timing-sensitive as you found — so we're treating this as corroboration rather than independent proof, and relying on your regression test for the interleaving. Thanks for the quick turnaround.