Re: [nodejs] ECONNRESET since update to Node 0.10

[Isaac Schlueter]

Without seeing anything except an error message, I can't really answer
that except to say that some net.Socket object is trying to read(),
and getting an ECONNRESET error. Someone sent someone else a RST
packet, and instead of ignoring it, Node is emitting an error because
of it, like it ought to have in the first place.

Maybe share more of your code, put error handlers on your client and
server connections, and use domain objects to get more context in the
error. That'd all help track down where this is coming from.


On Fri, Apr 5, 2013 at 9:19 AM, Jochen Delabie <jochen...@gmail.com> wrote:
> Since we updated from Node 0.8 to Node 0.10 we're seeing these errors:
>
> { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET',
> syscall: 'read' }
> Error: read ECONNRESET
> at errnoException (net.js:878:11)
> at TCP.onread (net.js:539:19)
>
>
> I know this error was previously ignored before 0.10, but I'd like to know
> what's causing this and how I can fix it/debug it?
> Is it a problem on our end (server) closing connections too soon?
>
> Thanks!
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jimb Esser]

As an extra data point, I've been seeing these too, since updating to node 0.10, but have no idea where they're coming from.  I think we wrap all of our HTTP requests such that errors get caught, but this is bubbling up to process.on('uncaughtException').  We also have warnings that fire if we ever get a request and do not respond to it, and those warnings are not firing when this occurs, so, other than the uncaughtException, we're not seeing anything that would indicate anything going wrong.  Admittedly, this is inside of a rather large and complex app, so it's quite possible we're missing something, but it's definitely not coming from our usual suspects for HTTP requests (Amazon APIs).

[Stuart]

I too have been seeing ECONNRESET. In my case, the ECONNRESET comes from a server that sends no data to the socket, does nothing but send a FIN packet. As I understand TCP, while this isn't ordinary, it isn't illegal, nor is it a reset. Since the server didn't send a RST packet, but a FIN, shouldn't the socket close report an end event?

[Ben Noordhuis]

On Tue, May 21, 2013 at 3:58 AM, Stuart <stu...@yellowhelium.com> wrote:
> I too have been seeing ECONNRESET. In my case, the ECONNRESET comes from a
> server that sends no data to the socket, does nothing but send a FIN packet.
> As I understand TCP, while this isn't ordinary, it isn't illegal, nor is it
> a reset. Since the server didn't send a RST packet, but a FIN, shouldn't the
> socket close report an end event?

The ECONNRESET is what is reported by the operating system so
evidently it thinks the connection has been reset.

If, for example, the peer sends a FIN before the connection has been
fully established (SYN sent but no ACK received yet), then most
operating systems will consider that a reset. ECONNABORTED would have
made more sense IMO but it is what it is.

[Matt Zukowski]

Pretty much in the same boat here. Occasionally seeing the following error, crashing the whole app. It's mostly unpredictable, but I can sometimes reproduce it by pounding the server with ab for a while. It never happens while the server is processing requests, only about 1 - 2 minutes after, when no requests are coming though:

events.js:72

        throw er; // Unhandled 'error' event

              ^

Error: read ECONNRESET

    at errnoException (net.js:884:11)

    at TCP.onread (net.js:539:19)

[Matt Zukowski]

Turns out my problem was caused by stray RabbitMQ connections getting closed. Installing the longjohn module helped debug this with better stack traces.

[radus...@ymail.com]

+1. Having this issue in production. Cannot reproduce on my testing env. I get it for both read and write. How can I supress it? 

[Jonahss]

I was running into this error and can reproduce it on version 0.10 but can no longer reproduce is on version 0.11.

So the source of the error may have been incidentally fixed as part of something else.

[Sam Roberts]

On Sun, Jun 30, 2013 at 11:35 AM, Jonahss <jon...@gmail.com> wrote:
>> I know this error was previously ignored before 0.10, but I'd like to know
>> what's causing this and how I can fix it/debug it?
>> Is it a problem on our end (server) closing connections too soon?

Wireshark the connection, its usually pretty obvious what about the
TCP exchange pattern is causing a reset when you see the packets.

One common cause is timing, the server might be doing a close()
instead of a shutdown(WRITE) (.end() in node), and the clients data
arrives after the close sometimes, and before at other times. If its
after, the connection is reset by the stack. Small changes in ordering
of operations (node's write of buffers, o/s scheduling when client and
server are running on the same system, system load, etc. can cause
what is essentially a bug in the use of TCP to appear or go away).

[Mike Gradek]

We had a similar issue, upgrading from 0.8 to 0.10, but the root cause was muddied by upgrades to modules as well. 

The issue occurred in our test suite, only when we ran a "large" number of tests. Running each test suite independently did not repro the error.

In the end, here's how we fixed the issue:

Analysis:

- Ran wireshark during a faulty test run, our tests issued HTTP requests to a local node server

- Checked the wireshark trace and noticed

  - a RST getting sent just before the test failure

  - many, many, many different port numbers being used for issuing the requests

Test hypothesis:

The variety of different ports being used in the trace led us to believe we had too many fd open

- we ran a counter on the number of requests made in our tests, and they would fail when we hit roughly the 250 mark

- we bumped up the fd limit locally, and ran the tests again, and they'd pass

Fix:

We used mocha and supertest to issue these requests, and realized that we were actually spinning up new port bindings on every request instead of reusing existing bindings.

So code that was written like so:

 var request = require('supertest');

 var app = require('../app');

 request(app).get(...);
 request(app).get(...);

Became

 var request = require('supertest');

 var app = require('../app');

 var supertest = request(app);

 supertest.get(...);
 supertest.get(...);

That solved the issue for us. 

In summary, during the upgrade, we ended up using many more file descriptors than we'd anticipate due to our test setup, and the OS started killing them causing the RST to get sent. We fixed up our tests and haven't seen the RST since.