ECONNRESET when running "many" tests

Hey there,

we run into random ECONNRESET problems for weeks now and tried many things to mitigate the problem. We use supertest for e2e testing a GraphQL API built with Nest.js.

We run it like so:

GraphQLQuery.ts:

export class GraphQlQuery {
    private static _bindings: Map<string, supertest.SuperTest<supertest.Test>>;

    static queryGraphQl(app: INestApplication, query: GraphQlQuery, token?: string): supertest.Test {
        if (!GraphQlQuery._bindings) {
            GraphQlQuery._bindings = new Map();
        }

        let binding = GraphQlQuery._bindings.get(app.getHttpServer().testHash);
        if (!binding) {
            binding = request(app.getHttpServer());
            GraphQlQuery._bindings.set(app.getHttpServer().testHash, binding);
        }

        const req = binding.post("/graphql");

        if (token) {
            req.set({ Authorization: `Bearer ${token}` });
        }

        req.send({ operationName: null, query: query.build() })
            .expect((response) => {
                if (!response.ok) {
                    console.error(JSON.stringify(response, null, 2));
                }
            })
            .expect(200);

        return req;
    }

    // ...
}

Somewhere in the tests, we can then do

GraphQlQuery.queryGraphQl(app, query, jwt).then((response) => {
    expect(response).toBe(something);
});

The reason we do all the GraphQLQuery._bindings-stuff is because we are aware that those bindings should be reused. We create the Nest.js app in our beforeAll()-hooks, where we also add the testHash to the HTTP Server. It's basically a UUID to ensure that we can map bindings to test-suites. We also run Jest with --runInBand on the CI, so nothing should run concurrently. We have around 80 tests now which all rely on supertest and the more tests we build, the worse it gets. Up until to a point where we need to run the whole CI-step multiple times until ECONNRESET doesn't happen any more. Sometimes up to 10 times or so, with each pipeline needing ~5 minutes.

Unfortunately I didn't find much information about our problems and since Nest.js recommends supertest (and I think I can say that on behalf of the team, we really like the approach), I am wondering what we're missing.

Any help appreciated!

Any update here? I seem to be getting a similar error:

Error: ECONNRESET: Connection reset by peer
      at Test.assert (./node_modules/supertest/lib/test.js:193:18)
      at localAssert (./node_modules/supertest/lib/test.js:159:12)
      at ./node_modules/supertest/lib/test.js:156:5

I wrote test to reproduce problem

describe('a lot of calls', function () {
  this.timeout(10000)

  it('should handle many requests', (done) => {
    const app = express();
    app.get('/', (req, res) => {
      res.send('hi');
    })

    const agent = request(app)

    const reqFn = (i) => {
      return new Promise((resolve, reject) => {
        const st = setTimeout(() => {
          clearTimeout(st)
          const req = agent.get('/')
          req.end((err, res) => {
            if (err) {
              console.log(i, err.message);
              reject(err)
            } else {
              res.text.should.be.eql('hi');
              resolve()
            }
          });
        }, 100)
      })
    }
    const promises = []
    for (let i = 0; i < 200; i++) {
      promises.push(reqFn(i))
    }
    Promise.all(promises)
      .then(() => done())
      .catch(err => done(err))
  });
});

There are may be different output in terminal after executions:

...
160 ECONNREFUSED: Connection refused
159 ECONNREFUSED: Connection refused

or

...
133 ECONNREFUSED: Connection refused
132 ECONNREFUSED: Connection refused

etc.

But i have no idea how to fix this :(

@Dezzpil

@Dezzpil thanks a lot! But I'm not quite sure if that directly relates to our problem, since we only see ECONNRESET and never ECONNREFUSED :/

same error with me. ECONNRESET is showing when running many tests, but It does not show when running the same test individually. In my case, I am not using Nest

It seems that when calling supertest many times in a concurrent manner this code https://github.com/visionmedia/supertest/blob/28116f9060c1904a433e7d69f1c14db32cf60279/lib/test.js#L61
will spin up multiple HTTP servers or start listening on multiple ports (?).

My solution is to call app.getHttpServer().listen(0); after test app initialization in the beforeAll hook. This way the HTTP server can warm up and all supertest calls use the same address.

So far worked fine for me. Hope that helps in your case as well.

@shadowgate15

Another issue that @shadowgate15 @spence-s and I found was that port 5000 is being used by AirPlay receiver on macOS Monterey via https://developer.apple.com/forums/thread/682332. I believe that @shadowgate15 is changing tests to use get-port instead to launch port for ZUUL / server.listen.

@niftylettuce

Good find @niftylettuce - that might be it.

Perhaps we should find a solution to avoid using port 5000, if that is the case?

@niftylettuce

@niftylettuce I did try to disable AirPlay Receiver, but still getting random "Connection reset by peer"

@niftylettuce

We need to fix the tests to use beforeEach and afterEach to make sure the app is started on a random port and closes the app after the test.

This issue should be reopened @niftylettuce.

And assign it to me I'll fix it later on this week.

@shadowgate15

@shadowgate15 I believe the ticket was never reopened; however, still a relevant issue to tackle! 😊

PR welcome

This is happening to me with Node v20 only.

Look at the evidence of this PR loopbackio/loopback-next#9751.

What solutions can I adopt?

UPDATE:

It seems that removing the for and using es6 syntax works...

-> FOR TEST v20 https://github.com/loopbackio/loopback-next/actions/runs/5745322969/job/15573163172?pr=9751
-> ES6 TEST v20 https://github.com/loopbackio/loopback-next/actions/runs/5745513122/job/15573621754?pr=9751

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.

It was reverted in Node 18.18.1 but never in 20.x 😢

We are experiencing the same issue with the latest node.js v20.9.0 version. Same for the 20.8.1 version.
The libuv seems to be the 1.46.0.
So, for now are there any workaround? Without considering an older version of node.

What worked for us is:

-const request = supertest(server);
+const request = supertest(`http://127.0.0.1:${server.address().port}`);

I couldn't figure out the exact reason but I suspected it was attempting to start another server so providing a URI instead somehow just fixed it.

Hopefully it's useful for at least one person going through this difficult and painful time. 😅

test: fix ECONRESET for metric and history

It was reverted in Node 18.18.1 but never in 20.x 😢

Really upsetting.

To run my tests in Node.js 20.11.0, I had to replace

server = await supertest(app)
response = await server.get(path)

with

server = await app.listen()
port = server.address().port
response = await fetch(`http://localhost:${port}${path}`)

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.

It was reverted in Node 18.18.1 but never in 20.x 😢

Hey) Did you have a chance to find out the root cause of the problem that is in the libuv?

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.
It was reverted in Node 18.18.1 but never in 20.x 😢

Hey) Did you have a chance to find out the root cause of the problem that is in the libuv?

Node 18.18.1 reverts libuv change which is about importing io_uring and doesn't add it back again, but Node 20.x doesn't.
Using Axios to do parallel http requests has no this problem. I think it shouldn't be duty of Node or libuv. They are chaning, but Supertest doesn't.

We also experienced the same issue, although we found out the ECONNRESET happened to us when upgrading to Node20, in combination of calling supertest in a Promise.all array of promises.

Untangling the Promise.all call and awaiting individually, solved the problem for us.

I encountered similar issues and resolved them by modifying the code as follows (node version: v20.14.0):

Before:

let app: ReturnType<typeof express>
let test: TestAgent

beforeEach(() => {
    app = express()
    // Middleware setup, etc.
    test = supertest(app)
})

After:

let app: ReturnType<typeof express>
let test: TestAgent
let server: ReturnType<typeof app.listen>

beforeEach(() => {
    app = express()
    // Middleware setup, etc.
    server = app.listen(0)
    test = supertest(server)
})

afterEach(() => {
    server.close()
})

It appears that passing the server to supertest instead of the app itself resolves the issue.

This looks like it was introduced in Node with the libuv@>=1.45.0 upgrade. It was introduced in Node 20.3.0 and Node 18.18.0.
It was reverted in Node 18.18.1 but never in 20.x 😢

Hey) Did you have a chance to find out the root cause of the problem that is in the libuv?

Node 18.18.1 reverts libuv change which is about importing io_uring and doesn't add it back again, but Node 20.x doesn't. Using Axios to do parallel http requests has no this problem. I think it shouldn't be duty of Node or libuv. They are chaning, but Supertest doesn't.

Setting parallel fixed my issue, even though I was not using supertest, but got similar issue with cucumberjs

fix: avoid server re-use race conditions

v7.1.0 released to npm – please try it out and let us know if it works for you 🎉

https://github.com/ladjs/supertest/releases/tag/v7.1.0

Thank you for your contributions 🙏

Our efforts to maintain this project are through our work on https://forwardemail.net (@forwardemail)

Looks like its resolved the issue for us - thank you!

Somehow I've seen this issue resurface (without any environment changes I'm aware of) - though seems like it works up to 5 simultaneous connections instead of the reported 3 in the PR. Happy to debug further if there's any suggestions for where to look

Navigation Menu

Repository navigation

ECONNRESET when running "many" tests #709

UPDATE:

8 remaining items

Before:

After:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ECONNRESET when running "many" tests #709

Description

Activity

jordancalder commented on Mar 2, 2021

Dezzpil commented on Mar 4, 2021

schw4rzlicht commented on Mar 4, 2021

lucasraziel commented on Oct 2, 2021

cmwd commented on Jan 4, 2022

niftylettuce commented on Jan 4, 2022

simplenotezy commented on Mar 30, 2022

simplenotezy commented on Mar 30, 2022

shadowgate15 commented on Mar 30, 2022

shadowgate15 commented on Mar 30, 2022

simplenotezy commented on Aug 28, 2022

titanism commented on Aug 29, 2022

frbuceta commented on Aug 3, 2023

UPDATE:

8 remaining items

Drew-Kimberly commented on Nov 2, 2023

luiof commented on Nov 8, 2023

eddyw commented on Nov 8, 2023

aercolino-cognizant-cepsa commented on Jan 30, 2024

vbuzivskoy commented on Feb 21, 2024

zyf0330 commented on Mar 5, 2024

Marcosaurios commented on May 2, 2024

nodify-at commented on Jun 19, 2024

Before:

After:

grumpy-huskie commented on Dec 4, 2024

titanism commented on Mar 20, 2025

luxaritas commented on Mar 20, 2025

luxaritas commented on Jun 16, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions