Http client hangs due to connection pooling

[Andrey]

Hi,

I'm writing non-blocking proxy server and faced with the issue: after several benchmark iterations my server stop handling requests. As I see pooled connections are in CLOSE_WAIT state (according to netstat) which means that remote server has closed them, but vert.x http client still keeps them for sending data. I suppose when new request arrives vert.x acquires one of them to pass the request and then hangs.

Can someone expose some thoughts on this?

My environment:

Vert.x: 3.1.0

JDK: 1.8

Thanks.   

[Nat]

Do you have a small repro?

[Andrey Zhuchkov]

val clientOptions =
                HttpClientOptions()
                        .setDefaultHost("$BUCKET.s3.amazonaws.com")
                        .setMaxPoolSize(1000)

        val client = vertx.createHttpClient(clientOptions)

        val serverOptions =
                HttpServerOptions()
                .setAcceptBacklog(1000)

        vertx.createHttpServer(serverOptions)
                .requestHandler { request ->
                    val path = request.path()

                    when (request.method()) {                        
                        HttpMethod.GET -> {
                            client.get(path) { s3Response ->
                                if (s3Response.statusCode() != 200)
                                    request.response().setStatusCode(s3Response.statusCode()).end()
                                else {
                                    val response =
                                            request.response()
                                                    .putHeader(LENGTH_HEADER, s3Response.getHeader(LENGTH_HEADER))

                                    val pump = Pump.pump(s3Response, response)

                                    s3Response.endHandler { response.end() }

                                    pump.start()
                                }
                            }
                            .putHeader("Date", "...") // put required headers
                            .end()
                        }
                        else ->
                            request.response().setStatusCode(400).end("Unsupported HTTP verb")
                    }
                }
                .listen(8080) { result ->
                    if (result.succeeded())
                        future.complete()
                    else
                        future.fail(result.cause())
                }

 run from main method: Launcher.main(arrayOf("run", MyVerticle::class.java.name))

It's a Kotlin if you want to run it. 

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/77_JUc3mEuY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/9c2ae20a-ff39-4b04-a9f3-548a7bc97706%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrey Zhuchkov

[Alexander Lehmann]

I wrote a small project a while back and I noticed that the handling of Connection: keep-alive is not completely correct in vert.x client (i.e. if the server decides the connection should be closed (and sends Connection: close), the pool does not notice that).
You could work around that by replacing Connection: keep-alive b Connection: close and disable keepAlive handling in the client.

We have talked about this issue before on the list, but I didn't get around to actually turning that into a proper bug report yet.

[Julien Viet]

there have been a couple of fixes recently but they were making the test suite failing.

that’s something to definitely improve, as usual providing a proper bug report is the hardest.

-- 
Julien Viet
www.julienviet.com

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/089aec15-14ec-45a0-a2bf-301c6d7abc2d%40googlegroups.com.

[Andrey Zhuchkov]

It seems that disabling Keep-Live solves the problem, but it decreases overall throughput drastically - from 2500 op/sec to 70 op/sec. Not sure why this happens but Apache Benchmark (ab) shows for the same server 2300 op/sec w/o keep-alive and 2700 op/sec w/ (-k option).

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/77_JUc3mEuY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/etPan.566f445a.4516dde9.10e%40juliens-mbp-2.

For more options, visit https://groups.google.com/d/optout.

--

Andrey Zhuchkov

[Alexander Lehmann]

Keep-alive will reduce the overhead, especially when using https, so the rate will be much lower when it's not used.

You could try to close the client connection at the end of the request if the header does not contain Connection: keep-alive, but I'm not sure if the connection is accessible from the http client.

I will write a bug reproducer when I have some time, for now you can only do a makeshift fix.

[Andrey Zhuchkov]

I filed the issue here - https://github.com/eclipse/vert.x/issues/1244. Demo project written in Java is attached.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d7f9f8e5-0abc-4379-8b29-8c7012e73203%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrey Zhuchkov

[Nat]

Can you replicate the problem with Vert.x 3.2.0?

[Andrey Zhuchkov]

Yep, sample project uses 3.2.0. already. :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/7cbb7f4b-afe9-45b7-b630-c2cd9a1af9f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrey Zhuchkov

[Andrey Zhuchkov]

It seems that root cause is using Pump. If I replace it with simple handler() (like in this example https://github.com/vert-x/vertx-examples/blob/master/src/raw/java/proxy/ProxyServer.java) the problem is gone, but I still cannot make to work Pump properly. Can someone look at https://gist.github.com/azhuchkov/155dbd132f57c37dc15d and say what i should add/change to prevent hanging due to source pausing by pump?

Another question is whether it needed to add endHandler BEFORE starting Pump or not? I've seen both of the examples in vertx-core manual.

Thanks.

--

Andrey Zhuchkov

[Alexander Lehmann]

I can take a look at the project this evening (or over the weekend), I have implemented something with Pump where it was important to pause the request before setting up the Pump and the handlers and then starting the Pump or something similar.

[Nat]

Btw, this issue might be related to https://groups.google.com/forum/#!topic/vertx/5-D1_gQzYqM.