How to troubleshoot ECONNRESET

[Przemo T]

Hi guys,

I'm working on HTTP proxy written in vertx and I see that my NodeJs client that calls the proxy, receives from time to time ECONNRESET errors. My understanding is that this error is received on the client when server ends connection on its end (I don't know too much details of TCP though).

First thing that my vertx proxy is doing is that it logs incoming requests but for those that client received ECONNRESET I don't see any logs on server side. Any ideas what may be wrong?

My client generates quite a heavy load. It starts with 100 concurrent requests up to 14000 and uploads files. But I don't see any specific pattern when that error occurs.

[Alexander Lehmann]

If you see the error on the client, the server has closed the connection when the client was writing data, which in the case of HTTP would be the request. Either that means that the connection was closed immediately after it was established or when using keep-alive the server closed the connection after sending a response back which the client wasn't expecting due to header lines in the response or after a timeout.

You may be able to analyse the issue with Wireshark however when it only happens some times and a lot of connections are correct, the amount of data to search through is probably a problem. If you can find connections that get connection reset in Wireshark, it should be possible to check these connections.

[Julien Viet]

well put :-)

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/1edba228-d806-4e12-b2cb-685233cb3dbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Lehmann]

If the issue is related to keep-alive it might be good to first check if the issue disappears when disabling keep-alive on the client.

[Przemo T]

Both apps are in AWS VMs. Client calls server machine by its public ip (not load balancer). Server is launched in docker.

Initially I tried to make no limit calls, so potentially there were 14k concurrent calls to server.

Next I limited requests to max 3k batches (next batch is started after previous one completes).

Still was seeing: {"code":"ECONNRESET","errno":"ECONNRESET","syscall":"read"} in NodeJS.

Next I added 'keepAlive: true, and maxSockets: 200' to node's http agent. That improved total processing time and reduced/eliminated ECONNRESET error. However setting maxSockets size to big numer like 1000 does not improve total time but increases number of ECONNRESET which I can't explain/understand.

I'm not sure if the error occurs in OS/network layer or if it is related to vertx/java or nodejs. ECONNRESET means that server endpoint did reset connection. Is vertx closing connections? Is there any vertx limit for number of opened incoming connections in httpserver (for those lost connections I'm pretty sure that my http server handler was not invoked at all so maybe this is OS limit?)

[Alexander Lehmann]

If the http server runs into a limit, it will probably stop accepting connections and not close them so you should see timeouts in this case. (In theory it would be valid for the http server to close the connection directly when it has accepted it, forcing the client to connect again, but that is not usually used in most http servers and i think vert.x doesn't either).

Could you please try keepAlive: false to check if that fixes the issue (but this will reduce the throughput due to more connections being opened I think)?