httpClient keepAlive - connection reuse

[ad...@cs.miami.edu]

Hi,

In the docs I see this statement:

For pooling to occur, keep-alive must be true on the HttpClientOptions (default is true). In this case connections will be pooled and re-used if there are pending HTTP requests waiting to get a connection, otherwise they will be closed.

I would expect that the keep-alive would keep the connection alive for a period of time even if there are no pending HTTP requests.  But this seems to close the connection immediately if there is not an immediate "next" request to be made.  I thought the whole idea of a keep alive is to keep a connection alive for a short period of time "in-between" requests.  But, this does not seem to be the case in the vert.x Http Client.  Am I understanding the docs correctly?

Thanks,

-Adam

[Julien Viet]

I think this is incorrect, connections are kept in the pool until they are closed by the server or reused.

there is also an on-going discussion in another thread about adding a timeout on these connections to close them when the timeout fires.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/173e9017-9890-44e6-b1f2-332cc5cb74ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ad...@cs.miami.edu]

That makes sense.  So, I think the docs should read something like this:

For pooling to occur, keep-alive must be true on the HttpClientOptions (default is true). In this case connections will be pooled and re-used so long as the connection is still alive. The connection will stay alive and re-pooled until it is closed by the server.

[Julien Viet]

yes it should be corrected, can you make a PR ?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/6229243a-30be-4066-9684-a6102070c933%40googlegroups.com.

[Tim Fox]

That's how it _used_ to work (things may have changed since then in the implementation).

The reasoning is simple: The point of keep alive is to reuse connections for better performance, but if you're sending requests so slowly there are no pending requests (i.e. requests go to the server and a response comes back before you send the next request) then you're going to obtain little or no performance improvement from using keep alive, so you may as well close the connection.

BTW this is exactly the same behaviour as the node.js http client pool, from which the vert.x one was originally modelled.

[Tim Fox]

Note, I'm not saying that's the best way to implement it, just how it used to implemented hence the comment :)

[Alexander Lehmann]

The keep-alive is usually a tradeoff between keeping the connections around longer based on a guess how often connections are required or closing them to avoid keeping too many open handles around.

Browsers will keep the connections open for some time and the server can decide a timeout as well (this may be 5 or 15 seconds in apache for example).

I think the current implementation uses the timeout of the server (by just dropping the connection when the server closes it) and otherwise keep the connection open indefinitely (at least I think so, I have to check the code)

The simplest example to show that a client side keep alive can be beneficial is something like the following

while(true) {
  sleep(2);
  get("/status", ...);
}

when using a keepalive that only reuses connections when there are some pending will close the connection each time the get is finished and open a new one for the next request. Since the keep-alive timeout is probably at least 5 seconds if the connections are kept open between calls, it will all use just one tcp connection saving the overhead of routing and ssl.

[Julien Viet]

I don’t remember having changed the behavior of this when I contributed to the client.

That being said both behavior are desirable, you might want to keep a connection a little in the pool before closing it to avoid recreating the connection but you want also to save resources if you don’t use them anymore.

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e9d08547-c01e-4cbc-9c4e-e0c5a255f1e1%40googlegroups.com.

[Tim Fox]

On Wednesday, 10 May 2017 12:00:24 UTC+1, Alexander Lehmann wrote:

The keep-alive is usually a tradeoff between keeping the connections around longer based on a guess how often connections are required or closing them to avoid keeping too many open handles around.

Browsers will keep the connections open for some time and the server can decide a timeout as well (this may be 5 or 15 seconds in apache for example).

I think the current implementation uses the timeout of the server (by just dropping the connection when the server closes it) and otherwise keep the connection open indefinitely (at least I think so, I have to check the code)

The simplest example to show that a client side keep alive can be beneficial is something like the following

while(true) {
  sleep(2);
  get("/status", ...);
}

when using a keepalive that only reuses connections when there are some pending will close the connection each time the get is finished and open a new one for the next request. Since the keep-alive timeout is probably at least 5 seconds if the connections are kept open between calls, it will all use just one tcp connection saving the overhead of routing and ssl.

But I'm not sure it matters for most cases. Think of it is this way (this is for straight TCP, for SSL there's more overhead in connection setup)

With the old behaviour, let's consider the cases.

Let network round trip time = T (maybe up to a few 100 of ms depending on route)

Let server connection timeout = S (500ms is common default)

Let t is the time between sending requests at the client

Case 1: "Fast sending", t < T. (E.g. if if T = 200 ms, then if you send more than 5 requests / second this will be "fast"), then connection will never be closed by client as there's always a pending request.

Case 2: Time t between sending requests greater equal than T but less than S, T >= t < S. In this case connection will be closed by the client after each request

Case 3: Time t between sending >= S. In this case all connection are closed by the server after one request and server timeout.

So the only difference in closing behaviour between old behaviour and new is if you send at a "medium" rate (e.g. between every 200ms and 5000ms), in this case you would incur a little bit of extra latency due to the connection setup. Extra latency for connection setup should also be equal T because a SYN needs to travel to server and an ACK needs to be returned.

So.. although old behaviour and new behaviour seem quite different, in most cases there's probably no difference in performance at all, unless you happen to hit that "sweet spot" between T and S when you'll incur a little bit of extra latency. I suspect this would be quite unusual. Most APIs are either heavily or used infrequently.

[Tim Fox]

On Wednesday, 10 May 2017 12:26:33 UTC+1, Tim Fox wrote:

On Wednesday, 10 May 2017 12:00:24 UTC+1, Alexander Lehmann wrote:

The keep-alive is usually a tradeoff between keeping the connections around longer based on a guess how often connections are required or closing them to avoid keeping too many open handles around.

Browsers will keep the connections open for some time and the server can decide a timeout as well (this may be 5 or 15 seconds in apache for example).

I think the current implementation uses the timeout of the server (by just dropping the connection when the server closes it) and otherwise keep the connection open indefinitely (at least I think so, I have to check the code)

The simplest example to show that a client side keep alive can be beneficial is something like the following

while(true) {
  sleep(2);
  get("/status", ...);
}

when using a keepalive that only reuses connections when there are some pending will close the connection each time the get is finished and open a new one for the next request. Since the keep-alive timeout is probably at least 5 seconds if the connections are kept open between calls, it will all use just one tcp connection saving the overhead of routing and ssl.

But I'm not sure it matters for most cases. Think of it is this way (this is for straight TCP, for SSL there's more overhead in connection setup)

With the old behaviour, let's consider the cases.

Let network round trip time = T (maybe up to a few 100 of ms depending on route)

Let server connection timeout = S (500ms is common default)

^^ should say 5000ms

[Tim Fox]

On Wednesday, 10 May 2017 12:19:22 UTC+1, Julien Viet wrote:

I don’t remember having changed the behavior of this when I contributed to the client.

I think it might have been changed before that, around Vert.x 2 time-frame.

 

That being said both behavior are desirable, you might want to keep a connection a little in the pool before closing it to avoid recreating the connection but you want also to save resources if you don’t use them anymore.

On May 10, 2017, at 12:05 PM, Tim Fox <timv...@gmail.com> wrote:

Note, I'm not saying that's the best way to implement it, just how it used to implemented hence the comment :)

On Wednesday, 10 May 2017 09:43:35 UTC+1, Tim Fox wrote:

That's how it _used_ to work (things may have changed since then in the implementation).

The reasoning is simple: The point of keep alive is to reuse connections for better performance, but if you're sending requests so slowly there are no pending requests (i.e. requests go to the server and a response comes back before you send the next request) then you're going to obtain little or no performance improvement from using keep alive, so you may as well close the connection.

BTW this is exactly the same behaviour as the node.js http client pool, from which the vert.x one was originally modelled.

On Tuesday, 9 May 2017 21:10:16 UTC+1, ad...@cs.miami.edu  wrote:

Hi,

In the docs I see this statement:

For pooling to occur, keep-alive must be true on the HttpClientOptions (default is true). In this case connections will be pooled and re-used if there are pending HTTP requests waiting to get a connection, otherwise they will be closed.

I would expect that the keep-alive would keep the connection alive for a period of time even if there are no pending HTTP requests.  But this seems to close the connection immediately if there is not an immediate "next" request to be made.  I thought the whole idea of a keep alive is to keep a connection alive for a short period of time "in-between" requests.  But, this does not seem to be the case in the vert.x Http Client.  Am I understanding the docs correctly?

Thanks,

-Adam

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

[ad...@cs.miami.edu]

Hi group,

Thanks for all the insights and lively conversation regarding this post.  How the HttpClient "should" work is above my pay grade, but i think I remember the JDK's HttpURLConnection does keep the TCP connection alive in-between requests.  So, perhaps that is why I thought the behavior would be the same with vert.x.  though I am not suggesting that we model Vert.x's behavior based on what a JDK class provides out of the box  :)

-Adam

On Wednesday, May 10, 2017 at 8:00:23 AM UTC-4, Tim Fox wrote:

On Wednesday, 10 May 2017 12:19:22 UTC+1, Julien Viet wrote:

I don’t remember having changed the behavior of this when I contributed to the client.

I think it might have been changed before that, around Vert.x 2 time-frame.

 

That being said both behavior are desirable, you might want to keep a connection a little in the pool before closing it to avoid recreating the connection but you want also to save resources if you don’t use them anymore.

On May 10, 2017, at 12:05 PM, Tim Fox <timv...@gmail.com> wrote:

Note, I'm not saying that's the best way to implement it, just how it used to implemented hence the comment :)

On Wednesday, 10 May 2017 09:43:35 UTC+1, Tim Fox wrote:

That's how it _used_ to work (things may have changed since then in the implementation).

The reasoning is simple: The point of keep alive is to reuse connections for better performance, but if you're sending requests so slowly there are no pending requests (i.e. requests go to the server and a response comes back before you send the next request) then you're going to obtain little or no performance improvement from using keep alive, so you may as well close the connection.

BTW this is exactly the same behaviour as the node.js http client pool, from which the vert.x one was originally modelled.

On Tuesday, 9 May 2017 21:10:16 UTC+1, ad...@cs.miami.edu  wrote:

Hi,

In the docs I see this statement:

For pooling to occur, keep-alive must be true on the HttpClientOptions (default is true). In this case connections will be pooled and re-used if there are pending HTTP requests waiting to get a connection, otherwise they will be closed.

I would expect that the keep-alive would keep the connection alive for a period of time even if there are no pending HTTP requests.  But this seems to close the connection immediately if there is not an immediate "next" request to be made.  I thought the whole idea of a keep alive is to keep a connection alive for a short period of time "in-between" requests.  But, this does not seem to be the case in the vert.x Http Client.  Am I understanding the docs correctly?

Thanks,

-Adam

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Alexander Lehmann]

I may be confusing something about the current behaviour of the client. I will try to write a unit test showing the behaviour that could be improved with persistent keepalive connections.

[Alexander Lehmann]

Keep-alive is currently handled by the client by keeping the connection open to be reused later. I believe this was implemented with changes for pipelining and http2

https://gist.github.com/alexlehm/333405eccb14f57f0e252bb9509e9aaf

We could add a timeout for the client side though and maybe also an option to turn it into the old behaviour where only pending requests use keep-alive (which would be a timeout of 0)

[Julien Viet]

we could have the option to close them instead of having a timer (the implementation is trivial).

indeed adding a timeout does not seem to be extra necessary after all, the server should close them when they are not used anymore.

http/2 client connections would not be affected by this change I think.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f08bfc95-bd60-44fc-841f-6f0bfa0935bd%40googlegroups.com.

[Alexander Lehmann]

We can add a boolean property, that will be quite easy to implement. I was wondering if this should be called enablePersistentConnections or keepConnections or better something like closeImmediately.

In addition we may want to consider a timeout option for the http server, otherwise the connections will be open indefinitely when both the client and the server are implemented in vertx.

[ad...@cs.miami.edu]

I think the easist option is a boolean called keepOpen, which would be the default.  This will just keep the connection open until either the server closes it, or if the connection is removed from the pool (due newer connections expelling it from the pool, I suppose).

if keepOpen is false, then it uses the old behavior of only staying open if there is an immediate pending request on the same connection.

I think a timeout option on the vertx server makes sense.

-Adam

[Alexander Lehmann]

ok, we need a strategy for expelling idle connections, haven't thought of that. When the pool is filled, the oldest idle connect could be closed.

[ad...@cs.miami.edu]

yes, expelling least recently used connection should work.

[Tim Fox]

The server already has a timeout option (serveroptions.setIdleTimeout())

[Tim Fox]

Looking at the code, it seems it should already work on the client too.

HttpClientOptions.setIdleTimeout()

[Alexander Lehmann]

I assume this applies to a network timeout (when trying to read data but nothing arrives), not to the state where no data is expected since it is defined in TCPSSLOptions.

[Tim Fox]

It's the Netty idle timeout, the connection will be closed after timeout if there are no reads or writes on it.

Have you tried it?

[Alexander Lehmann]

Ok, that works, however that timeout is applied to all phases of the http protocol, which means that the client will also close the connection when the servers takes more than n seconds to produce a response, which is not what we want.

Assume that we set a idle timeout for network activity of 1 second and the server takes 10 seconds to create the result before sending anything, it will not work as the timeout applies to that as well

https://gist.github.com/alexlehm/2883ae691f5bc87bc03184fe6dc59466

[Julien Viet]

I agree with Alex that it is not the same.

For the sake of simplicity I am inclined to have an option that either keep the channel open (i.e what we have today) or just close it when the list of waiters is empty (like it used to be).

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/78ac69ef-0687-4db7-a0ad-51f071822e08%40googlegroups.com.

[Tim Fox]

On Sunday, 14 May 2017 12:46:25 UTC+1, Alexander Lehmann wrote:

Ok, that works, however that timeout is applied to all phases of the http protocol, which means that the client will also close the connection when the servers takes more than n seconds to produce a response, which is not what we want.

I guess I don't really understand what behaviour you want, perhaps you can describe in more detail?

[Alexander Lehmann]

I guess what is necessary is different timeouts in the different phases of the request or when the connection is idle.

This is not the same as a network timeout on the connection as this is applied to any phase of the connection.

For the server, this means that when a request is finished and the preconditions of keepalive are satisfied, the connection is kept open by the server until it is either closed by the client or a wait time has passed (in httpd apache that is 15 or 5 seconds, but that could be configurable). If the preconditions are not satisfied, the connection is closed immediately, e.g. when the client has sent a Connection: close header or uses HTTP/1.0.

For the client, this means that when a request is finished and the server has signified that keepalive is available, the connection is kept open until it is closed by the server or a wait time has passed or if the server sent a Connection header with a wait time limit, the client can close the connection when this time has expired (whichever is smaller).

None of these things are absolutely required, for example it would be ok to keep the client connection open until the server closes the connection (though that might use up the pool connections if connections to different servers are kept open indefinitely) or the old behaviour of using the keepalive only when requests are pending would be possible as well.

[Tim Fox]

On Monday, 15 May 2017 22:09:29 UTC+1, Alexander Lehmann wrote:

I guess what is necessary is different timeouts in the different phases of the request or when the connection is idle.

This is not the same as a network timeout on the connection as this is applied to any phase of the connection.

For the server, this means that when a request is finished and the preconditions of keepalive are satisfied, the connection is kept open by the server

I think you're overthinking things a bit - I think it's fine to just close the connection after idleTimeout() on the server side (this has always been the recommended way of doing things when writing http servers). In practice you'll have a request timeout in vertx-web too - just make sure the request timeout < idle timeout.

fwiw, I think node.js does pretty much the same thing here too.

 

until it is either closed by the client or a wait time has passed (in httpd apache that is 15 or 5 seconds, but that could be configurable). If the preconditions are not satisfied, the connection is closed immediately, e.g. when the client has sent a Connection: close header or uses HTTP/1.0.

For the client, this means that when a request is finished and the server has signified that keepalive is available, the connection is kept open until it is closed by the server or a wait time has passed or if the server sent a Connection header with a wait time limit, the client can close the connection when this time has expired (whichever is smaller).

Again, on the client, setting idleTimeout together with a request timeout which is smaller should do the trick.

 

None of these things are absolutely required, for example it would be ok to keep the client connection open until the server closes the connection (though that might use up the pool connections if connections to different servers are kept open indefinitely) or the old behaviour of using the keepalive only when requests are pending would be possible as well.

Personally, I think the current behaviour works ok (keeping the connection open at the client). If ppl want to close it sooner, an idleTimeout can be applied

[Alexander Lehmann]

Hi Julien,

if we want to implement an option to close the connections immediately when no pending requests or to keep them open in the client, do you want to have that in the next version?

I could maybe implement that Monday as that is a public holiday here.


bye, Alexander

On Thursday, May 11, 2017 at 8:13:04 AM UTC+2, Julien Viet wrote:

[Julien Viet]

Sure go ahead, sorry for late reply

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/29da754b-081c-401b-9733-cefb277bc9f6%40googlegroups.com.