Clarification on HttpClient pipelining and keep-alive

[N8]

Hi all, 

I came across an interesting bug in the project I am working on which reflects that I either don't understand how things are suposed to work, or they are not in fact working as desired. As a caveat, I am on slightly tweaked version of 1.2.3.final, so it's possible this issue has been addressed, and if so, I'll have to upgrade or backport the fixes. 

The problem: I want to reuse connections to a given host since requests to the same host will be made many times. These will be over SSL, so avoiding the handshake that comes with each new connection will have significant performance impact. However, when keep-alive is true (default) the same connection is used for many requests and I don't see how/when a new connection would ever get created. Lets say I make a request to a host, and this particular request takes a very long time to respond, and in the meantime, additional requests are being made using the same instance of HttpClient (to the same host). These additional requests, get the same connection out of the pool that was used for the first request. Each request gets added to the connection's request queue (requests member of ClientConnection), the request is written to the connection, and the connection returned to the pool thus making sure that it will be the connection that is used for the next request. The problem is that until the first request is completed, all subsequent requests made on the same connection are blocked and will not complete until the one before it has. This means if two requests are made, and the first takes 30 seconds, the second, however small, will take 30+ seconds to complete. 

I gleamed most of this running through the HttpClient and related code in a debugger. I had the boss threads and max pool size set to 50 (just to exaggerate). 

If I disable keep-alive, then each request get's it own connection and many requests can be made in parallel and they do not interfere with each other. 

So, my question is, if a connection already has outstanding requests, why does it not create a new connection to the handle a another request instead of putting subsequent requests on the same connection's queue? Under what conditions will new connections be created instead of pipelining (which I assume can only be serial). 

I am stuck here, as in my application some connections to the server may stream large files which then block other requests until the file is done streaming. 

Thanks for the thoughts on this. 

[Tim Fox]

If you're pipelining on a single connection it should be clear that responses have to be returned in the same order that requests were returned, therefore it's inevitable you have to wait for previous ones to complete before getting your response.

Pipelining is generally useful when request processing time << connection set up time.

If you have requests that take 30 seconds for you, pipelining doesn't seem to make a lot of sense.

When does a keep alive pool create a new connection?

Keep alive connections are taken from the pool when you start to write a request, and returned when the request is ended. So they don't stay out of the pool very long, this is why you're seeing all your requests using the same connection.

In the old days when a single pool was shared amongst many threads this meant new connections were created more frequently but since now a pool is local to a context it doesn't happen so much.

We could enhance the pool to return to instead create new connections if more than > N connections are pipe-lined, if that helped. Or you could just create N connection pools for now.  

[N8]

Thanks for the response. I think what I can do is to pool/pipeline requests I know will be small/short, and open a new connection for requests that I know have the potential to gum up the works. WHen you say create N connection pools, does that mean create N instances of HttpClient?

[Tim Fox]

On 05/12/12 18:23, N8 wrote:
> Thanks for the response. I think what I can do is to pool/pipeline
> requests I know will be small/short, and open a new connection for
> requests that I know have the potential to gum up the works. WHen you
> say create N connection pools, does that mean create N instances of
> HttpClient?

yep

> The problem is that until the first request is completed, *all

> subsequent requests made on the same connection are blocked

> *and will not complete until the one before it has. This means

> if two requests are made, and the first takes 30 seconds, the
> second, however small, will take 30+ seconds to complete.
>
> I gleamed most of this running through the HttpClient and
> related code in a debugger. I had the boss threads and max
> pool size set to 50 (just to exaggerate).
>
> If I disable keep-alive, then each request get's it own
> connection and many requests can be made in parallel and they
> do not interfere with each other.
>
> So, my question is, if a connection already has outstanding
> requests, why does it not create a new connection to the
> handle a another request instead of
> putting subsequent requests on the same connection's queue?
> Under what conditions will new connections be created instead
> of pipelining (which I assume can only be serial).
>
> I am stuck here, as in my application some connections to the
> server may stream large files which then block other requests
> until the file is done streaming.
>
> Thanks for the thoughts on this.
>
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit
> https://groups.google.com/d/msg/vertx/-/iFgixBm7qSkJ.
> To post to this group, send an email to ve...@googlegroups.com.
> To unsubscribe from this group, send email to
> vertx+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/vertx?hl=en-GB.


--
Tim Fox

Vert.x - effortless polyglot asynchronous application development
http://vertx.io
twitter:@timfox

[N8]

One final thought, it seems useful to use pooling without pipelining. In other words, each request gets it's own connection, but the connections come from a pool so if there is a free one available, that can be used. Any reason pipelining and keep-alive are married? What about separating them out?

If you think it's an idea, I'll try it out and make a pull request. 

[Tim Fox]

On 05/12/12 18:31, N8 wrote:
> One final thought, it seems useful to use pooling without pipelining.
> In other words, each request gets it's own connection, but the
> connections come from a pool so if there is a free one available, that
> can be used. Any reason pipelining and keep-alive are married? What
> about separating them out?

Can you elaborate? I'm not sure I follow.

> <https://groups.google.com/d/msg/vertx/-/iFgixBm7qSkJ>.

> > To post to this group, send an email to ve...@googlegroups.com

> <javascript:>.

> > To unsubscribe from this group, send email to

> > vertx+un...@googlegroups.com <javascript:>.

> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB

> <http://groups.google.com/group/vertx?hl=en-GB>.

>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous application development
> http://vertx.io
> twitter:@timfox
>

> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/vvqF-1W4ptkJ.

[N8]

Essentially what we have now, but with a mode that creates a new connection (up to max pool size) if all existing connections already have at least one request assigned to them. It could be if we hit max pool size, then we use a connection that already has a request on it (probably better to chose the one with the fewest number of pending requests). This would aggressively create connections up to the pool max, but would strive to give each request it's own dedicated connection, either an unoccupied one from the pool, or a new one that then gets added to the pool. 

[Tim Fox]

On 05/12/12 18:44, N8 wrote:
> Essentially what we have now, but with a mode that creates a new
> connection (up to max pool size) if all existing connections already
> have at least one request assigned to them. It could be if we hit max
> pool size, then we use a connection that already has a request on it
> (probably better to chose the one with the fewest number of pending
> requests). This would aggressively create connections up to the pool
> max, but would strive to give each request it's own dedicated
> connection, either an unoccupied one from the pool, or a new one that
> then gets added to the pool.

+1 That's doable.

> <https://groups.google.com/d/msg/vertx/-/vvqF-1W4ptkJ>.

> > To post to this group, send an email to ve...@googlegroups.com
> <javascript:>.
> > To unsubscribe from this group, send email to
> > vertx+un...@googlegroups.com <javascript:>.
> > For more options, visit this group at
> > http://groups.google.com/group/vertx?hl=en-GB
> <http://groups.google.com/group/vertx?hl=en-GB>.
>
>
> --
> Tim Fox
>
> Vert.x - effortless polyglot asynchronous application development
> http://vertx.io
> twitter:@timfox
>
> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To view this discussion on the web, visit

> https://groups.google.com/d/msg/vertx/-/2MIUFec22gAJ.

[Adam Hathcock]

"It could be if we hit max pool size, then we use a connection that already has a request on it (probably better to chose the one with the fewest number of pending requests)."

Just to clarify: You would want a queue for the pool and no enqueue requests on the connection.  Choosing the connection with the fewest number of requests on it would be worse if the connection is currently transferring a lot of data.  The connections should look at the pool queue if there is anything left to process if they quickly service the requests.

I guess making a pool queue would be more work as each connection has it's on queue for pipelining.

[Adam Hathcock]

To clarify my question: Why have pipelining when you have a connection pool?  

It seems pipelining queues up requests on a connection so you have a queue per connection.  The client would distribute requests to the connections regardless of request-response time.  It would be more efficient for the pool's waiters to be the single queue and connections only get a waiter when the request-response is completed.  This just means turning off pipelining would be the best thing to do.

[sANTo L]

I know this is a very old thread, but I have a similar issue and it seems nothing has changed since the start of this topic.

I want to be able to reuse HttpClient connections to avoid the overhead of creating a new connection with each request, as there will be a lot of requests coming in, all targeted to the same host.
This means that I have to set keepalive to true, but as far as I can tell this also seems to imply that pipelining is enabled and that's a problem in my case because the host that I am talking to supports keepalive but not pipelining of requests.
So whenever I enable keepalive, I receive a lot of "400 Bad Request" responses from the host because a previous response was not yet fetched by the client.

Therefore the suggestion from N8 about pooling without pipelining seems a nice way to work around servers that don't support the pipeling of requests.

Santo

[Norman Maurer]

If a server supports http 1.1 it must support pipelining snd keepalive. If not the server is defunkt

--

You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sANTo L]

Hi Norman,

I understand what you're saying, but nevertheless there are situations where it would be beneficial to be able to have keepalive enabled but pipelining turned off, even if the server supports pipelining.
Just look at the discussion in this thread.

So would it be possible to have an option to turn the pipelining off without affecting the keepalive ?

Thanks,

Santo

[Tim Fox]

On 02/05/14 21:55, sANTo L wrote:

I know this is a very old thread, but I have a similar issue and it seems nothing has changed since the start of this topic.

I want to be able to reuse HttpClient connections to avoid the overhead of creating a new connection with each request, as there will be a lot of requests coming in, all targeted to the same host.
This means that I have to set keepalive to true, but as far as I can tell this also seems to imply that pipelining is enabled and that's a problem in my case because the host that I am talking to supports keepalive but not pipelining of requests.
So whenever I enable keepalive, I receive a lot of "400 Bad Request" responses from the host because a previous response was not yet fetched by the client.

Why not just delay sending the next request until the previous response has been received?

Therefore the suggestion from N8 about pooling without pipelining seems a nice way to work around servers that don't support the pipeling of requests.

Santo

[sANTo L]

On Saturday, May 3, 2014 9:54:03 AM UTC+2, Tim Fox wrote:

On 02/05/14 21:55, sANTo L wrote:

I know this is a very old thread, but I have a similar issue and it seems nothing has changed since the start of this topic.

I want to be able to reuse HttpClient connections to avoid the overhead of creating a new connection with each request, as there will be a lot of requests coming in, all targeted to the same host.
This means that I have to set keepalive to true, but as far as I can tell this also seems to imply that pipelining is enabled and that's a problem in my case because the host that I am talking to supports keepalive but not pipelining of requests.
So whenever I enable keepalive, I receive a lot of "400 Bad Request" responses from the host because a previous response was not yet fetched by the client.

Why not just delay sending the next request until the previous response has been received?

Well, that's the whole point of the discussion: let vert.x keep a pool of connections and decide which request can use what connection.

To quote N8:

"One final thought, it seems useful to use pooling without pipelining. In other words, each request gets it's own connection, but the connections come from a pool so if there is a free one available, that can be used. Any reason pipelining and keep-alive are married? What about separating them out?"

And you answered with:
"+1 That's doable"

So I guess it is doable ;-)

Santo

[Jorge L.]

 I'm also concerned with mixing pool and pipelining. We are about to launch to production but this is a big issue. Our platform is basically a proxy to different backend services but we've found that there are some resources that may take 50ms, 10 seconds, or even 10 minutes. Pipelining is really nice when delays are small, but when responses are very slow, then pipeline will kill the average response time.

Imagine the scenario where 10 requests of 50ms are mixed with a connection of 10 minutes in the same pipeline. It will reply in the worst case and all the responses will be received in 10 minutes.

We could disable keepalive, but without a connection pool, it is inefficient (establishing the connection is not cheap) and it is very easy to exhaust ports (in TIME-WAIT status) when you try to have a high concurrency (imagine 1000 threads or simultaneous connections continuously opening new connections to our backend server).

So, is there any chance to separate pipelining from connection pooling? +1 for this divorce

[sANTo L]

Hi Jorge,

FYI: I created a BZ ticket for this: https://bugs.eclipse.org/bugs/show_bug.cgi?id=434402

regards,

Santo

[Jorge L.]

Thanks. For us, this issue is important.

I've been looking into the source code and I think that it's rather easy to disable pipelining. Soon I will provide a patch with a new option (pipelining=false) to disable pipelining. Default option should be true to get backwards compatibility. Let's see if the patch can be accepted.

[Jorge L.]

Finally I sent a patch to https://bugs.eclipse.org/bugs/show_bug.cgi?id=434402

[Norman Maurer]

I’m still not sure I understand the problem / concern… Why not just do what Tim suggested and only send the next request once you received the full response ?

-- 
Norman Maurer

--

[Jorge L.]

The only difference between enabling pipelining or not is to return the HTTP connection after the request is ended (pipelining) or after the response is received (no pipelining).

Enabling pipelining should be the rare case. According to HTTP1.1 specification (see bug): "Clients SHOULD NOT pipeline requests using non-idempotent methods or non-idempotent sequences of methods", but vertx is pipelining any HTTP method. It also creates a performance penalty when there is a substantial difference in response time in different requests.

I think Tim's proposal would create more complex code to build the HTTP client. I guess that I would have to create a queue of requests to send the next request after receiving a response. Perhaps I would also have to tackle with timeouts (to discard requests from the queue). And I'm not sure that it would work when instances>1. I mean, if we have 10 vertx instances but only 1 maxSocket, I can control the reception of responses for my instance, but the socket is shared by all the instances and we could not avoid sending more than 1 request simultaneously in the same socket.

If you could have a look at the patch, it is very simple and it maintains backwards compatibility to avoid any conflict with previous versions.

[sANTo L]

Hi Norman,

I think Jorge described it very well indeed.

The client should not pipeline all requests for several reasons (also see RFC2616) , while it is doing that at the moment.
And yes, it is possible to work around this, but it's just a workaround - with several drawbacks for the enduser - and therefore not a real solution.

Similar to saying that a HTTP 1.1 server is defunct when it doesn't support pipelining (see earlier response), I think a client is also kind of defunct when it pipelines all its requests without providing the option to disable it.

regards,

Santo

On Saturday, May 10, 2014 4:57:08 PM UTC+2, Norman Maurer wrote:

[Tim Fox]

On 10/05/14 19:39, sANTo L wrote:

Hi Norman,

I think Jorge described it very well indeed.

The client should not pipeline all requests for several reasons (also see RFC2616) , while it is doing that at the moment.
And yes, it is possible to work around this, but it's just a workaround - with several drawbacks for the enduser - and therefore not a real solution.

Similar to saying that a HTTP 1.1 server is defunct when it doesn't support pipelining (see earlier response), I think a client is also kind of defunct when it pipelines all its requests without providing the option to disable it.

You can disable pipelining - set keepAlive to false.

regards,

Santo

On Saturday, May 10, 2014 4:57:08 PM UTC+2, Norman Maurer wrote:

I’m still not sure I understand the problem / concern… Why not just do what Tim suggested and only send the next request once you received the full response ?

-- 
Norman Maurer

[Tim Fox]

Please don't send patches to BZ, we accept patches via GitHub pull requests.

--

[Tim Fox]

On 10/05/14 18:22, Jorge L. wrote:

The only difference between enabling pipelining or not is to return the HTTP connection after the request is ended (pipelining) or after the response is received (no pipelining).

Enabling pipelining should be the rare case.

Disagree. Idempotent method are GET, PUT, HEAD and DELETE. By far the most commonly used method is GET which is fine to pipeline.

In practice if you were doing a sequence of non-idempotent methods, e.g. POST something, then GET the same resource you would naturally do that as part of a workflow and would wait for the result of the POST before doing the GET anyway, i.e. common sense dictates you wouldn't pipeline it anyway.

I don't have an issue with separating keep alive and pipelining and I will look at your pull request. But I don't think it's the big issue you are making it out to be.

Currently, you can disable pipelining anyway by setting keep alive to false. Sure, a new connection will need to be opened for each request, but if you're not pipelining then you're going to have a network round trip *anyway*, and with most TCP setups when you close a connection the OS doesn't actually close it, it keeps it in TIME_WAIT state, so if you shortly after open a new TCP connection to the same port/host then it's likely it will reuse the "closed" connection - i.e. you won't have the connection setup costs.

But, like I say, I will take a look at the pull request when you submit it :)

According to HTTP1.1 specification (see bug): "Clients SHOULD NOT pipeline requests using non-idempotent methods or non-idempotent sequences of methods", but vertx is pipelining any HTTP method. It also creates a performance penalty when there is a substantial difference in response time in different requests.

I think Tim's proposal would create more complex code to build the HTTP client. I guess that I would have to create a queue of requests to send the next request after receiving a response. Perhaps I would also have to tackle with timeouts (to discard requests from the queue). And I'm not sure that it would work when instances>1. I mean, if we have 10 vertx instances but only 1 maxSocket, I can control the reception of responses for my instance, but the socket is shared by all the instances and we could not avoid sending more than 1 request simultaneously in the same socket.

If you could have a look at the patch, it is very simple and it maintains backwards compatibility to avoid any conflict with previous versions.

On Saturday, May 10, 2014 4:57:08 PM UTC+2, Norman Maurer wrote:

I’m still not sure I understand the problem / concern… Why not just do what Tim suggested and only send the next request once you received the full response ?

-- 
Norman Maurer

--

[Tim Fox]

On 09/05/14 15:24, Jorge L. wrote:

 I'm also concerned with mixing pool and pipelining. We are about to launch to production but this is a big issue. Our platform is basically a proxy to different backend services but we've found that there are some resources that may take 50ms, 10 seconds, or even 10 minutes. Pipelining is really nice when delays are small, but when responses are very slow, then pipeline will kill the average response time.

Imagine the scenario where 10 requests of 50ms are mixed with a connection of 10 minutes in the same pipeline. It will reply in the worst case and all the responses will be received in 10 minutes.

We could disable keepalive, but without a connection pool, it is inefficient (establishing the connection is not cheap) and it is very easy to exhaust ports (in TIME-WAIT status)

If you're exhausting ports it means you're not reusing connections. Reusing is exactly what you want to do in this case, and it will reduce the connection setup time.

when you try to have a high concurrency (imagine 1000 threads or simultaneous connections continuously opening new connections to our backend server).

So, is there any chance to separate pipelining from connection pooling? +1 for this divorce

On Sunday, May 4, 2014 9:42:45 PM UTC+2, sANTo L wrote:

On Saturday, May 3, 2014 9:54:03 AM UTC+2, Tim Fox wrote:

On 02/05/14 21:55, sANTo L wrote:

I know this is a very old thread, but I have a similar issue and it seems nothing has changed since the start of this topic.

I want to be able to reuse HttpClient connections to avoid the overhead of creating a new connection with each request, as there will be a lot of requests coming in, all targeted to the same host.
This means that I have to set keepalive to true, but as far as I can tell this also seems to imply that pipelining is enabled and that's a problem in my case because the host that I am talking to supports keepalive but not pipelining of requests.
So whenever I enable keepalive, I receive a lot of "400 Bad Request" responses from the host because a previous response was not yet fetched by the client.

Why not just delay sending the next request until the previous response has been received?

Well, that's the whole point of the discussion: let vert.x keep a pool of connections and decide which request can use what connection.

To quote N8:

"One final thought, it seems useful to use pooling without pipelining. In other words, each request gets it's own connection, but the connections come from a pool so if there is a free one available, that can be used. Any reason pipelining and keep-alive are married? What about separating them out?"

And you answered with:
"+1 That's doable"

So I guess it is doable ;-)

Santo

--

[Jorge L.]

The problem is that I need a connection pool for performance reasons and to avoid exhaustion of port by TIME-WAIT. But I want to disable HTTP pipelining because some responses are received in 50 ms, but other ones in 10 minutes. With pipelining, responses need to be received in correct sequence and it will make the slowest response delay the other responses. So in this scenario, if I send a request of 10 minutes, and 100 request of 50 ms, the average time of all of them would be 10 minutes with pipelining.

In your previous messages you have suggested both to set and unset keepAlive=true. But we want a different approach.

I understand the advantages of HTTP pipelining but our proposal is to offer the chance to disable it.

[Jorge L.]

I'll try to send a PR. When I tried it, I received a forbidden error.

[Tim Fox]

On 11/05/14 11:46, Jorge L. wrote:

The problem is that I need a connection pool for performance reasons and to avoid exhaustion of port by TIME-WAIT.

Have you tried setting your client to reuse addresses?

client.setReuseAddress(true)

This should eliminate exhaustion of ports due to too many connections in TIME_WAIT state, as closing connections will be reused and it should make connection setup time effectively zero. If you couple this setting keepAlive to false on the client, then you get pretty much what you want.

[Jorge L.]

No. I haven't played with it but I could give it a try.

[Jorge L.]

The PR is already available at: https://github.com/eclipse/vert.x/pull/831