Node.js client and ECONNRESET when querying elasticsearch

[Pepa Štefan]

Hi, I'm trying to query elasticsearch from node.js and from quite regularly I get error

[Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read'.

I query by scroll id. The queries goes one after other, quite quickly, one query takes maybe 10-30ms to resolve at server side. For querying I use request module and the requests should go in series (nothing parallel).

When I look at the requests through Fiddler, there is nothing interesting.

So I ran Process Monitor and Process Explorer. 

Process Explorer shows me that there is about 20-30 tcp connections, most of them are in fin_wait2 state. After some time they are probably closed (as they disappear).

Process Monitor shows me that when the process exists, there is a lot of requests that are disconnected. Last time that was 38 disconnected connections. I exit the process when I get ECONNRESET. In other words - if the process gets ECONNRESET, there are connections that are not disconnected.

Does the connections close ElasticSearch because there are some limits for open connections? No idea.

Anyway, I'd like to avoid ECONNERESET at all. Any ideas, please?

[Jimb Esser]

My experience with AWS, and most any cloud service, is that a given connection can fail for any reason at any time.  You need to retry requests if they get disconnected or any other transport-level error (and, probably most 50x errors as well, I know a bunch of the AWS services, if you are putting heavy load on them, return with "service busy, retry later" type errors, and their API docs always say to do an exponential backoff and retry if you get an error).

I think it is natural for closed connections to remain around in the fin_wait2 state, so I would not worry about those too much.  However, if you're doing requests in serial, ideally you want to be re-using a pooled connection, otherwise you're paying the (very expensive) HTTPS/SSL handshake cost with every request, and the fact that there are a bunch of closed connections means that pooling is definitely not happening.  I think node 0.12 handles this better than 0.6-0.10, and there should be some module to use with request (forever-agent, I think) which would let connection pooling work on any version of node.

Hope this helps!

  Jimb Esser