Existing connection forcibly closed issue in HTTP Client

[Manish Kumar]

Our application works as a HTTP proxy, and there are 1000s of client connected to our proxy, and our proxy route those requests to destination server.

We are using connection pooling(pool size is 30000) in a real high volume environment, and seeing following error in logs. We never saw these errors in load testing, 

and rarely saw these errors in a low volume environment. There are firewall and LB in front to destination server from our proxy app.

java.io.IOException: An existing connection was forcibly closed by the remote host

at sun.nio.ch.SocketDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(Unknown Source)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)

at sun.nio.ch.IOUtil.read(Unknown Source)

Is there something wrong I might be doing? I believe that it something on network, and out of my control on how I can fix this error. 

We definitely want to use connection pooling because of there could be upto 50000 clients connecting to my app.

[Tim Fox]

If this is occurring on the proxy and relates to the incoming connection then it could just be a client has forcibly closed a connection, which is not something under your control.

[Manish Kumar]

This is happening between our app and destination server.

Client (could be up to 50000 unique clients) . -> Our Vertx app acting a proxy -> Destination server

Error is happening on our app(proxy) where it's using HTTP Client to connect to destination server.

[Tim Fox]

Yes, I realise the error is happening in your proxy, but it doesn't tell you whether that relates to incoming or outgoing connections. You should be able to add some logging to give you that info.

[Manish Kumar]

Is this how I should caught connection errors of outgoing and incoming requests?

// For outgoing request

httpClientRequest.connectionHandler(conn -> {
    conn.exceptionHandler(ex -> {
        
    });
});

// For incoming request
httpServerRequest.connection().exceptionHandler(ex -> {
    
});

[Julien Viet]

for the HttpServerRequest you can set it on the HttpServerRequest itself.

setting it an handler on the connection is rather useful when you are using the HTTP/2 protocol and want to be aware of connection level failures

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b0ea16c7-b37b-43df-a210-60a31f125deb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manish Kumar]

I am setting connection handler for Http Client(outgoing request). I couldn't find a way to get hold of connection 

object from HTTP Client unless I attach the handler, so that I could attach "connection error handler" for outgoing request.

Sorry if I am being ignorant or missing something.

[Manish Kumar]

Julien,

Do you know if it's any good to attach the "connection error handler" on HttpClient (we aren't HTTP2)? This is what I am using, and see weird behavior.  I am using cports tool to kill connections.

clientRequest.connectionHandler(conn -> {
    conn.exceptionHandler(ex -> {
        // This executes intermittently. When this executes then "request level error handler" isn't executing.

        // I noticed that it's executing when we are killing keep-alive connection and that connection isn't serving any active request.
    });
});
clientRequest.toObservable().subscribe(response -> {
            
        },
        throwable -> {
            // This executes intermittently. When this executes then "connection level error handler" isn't executing.     

            // I noticed that it's executing when we are killing keep-alive connection that's serving active request.
        });

On Sunday, April 23, 2017 at 2:53:22 AM UTC-4, Julien Viet wrote:

[Julien Viet]

Hi,

the clientRequest connectionHandler should be called independently of the http client request handler.

if you can reproduce the issue in a small java program, I would be glad to look at the case.

that being said I don’t recommend to use the HttpClient with RxJava and rather use the WebClient if you can, it’s much more usable.

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ca17dc4a-863f-466e-8470-4fad79552705%40googlegroups.com.

[Manish Kumar]

We use HttpClient all over the place and it will take lot of time to replace it with WebClient. Are there any performance/deadlock concern with HttpClient? If no then we can take some breath and convert it to WebClient at our own pace.

[Julien Viet]

I understand.

The WebClient is based on HttpClient

as I said, if you can provide a clear reproducer someone (eventually me) would have a look at it.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f4241650-c1ca-44f9-8434-fe8196c43da4%40googlegroups.com.

[Manish Kumar]

Reproducer is definitely coming. But I need some time because I want to provide you everything in reproducer(including remote server as well) so that you can run it out-of-box.

[Julien Viet]

such reproducers are great!

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b2a18f44-a7a3-4da3-8b15-403d98745ff2%40googlegroups.com.