Fork (was Roadmap): HTTP client implementation

[Benjamin van der Veen]

On Wed, Aug 10, 2011 at 3:11 PM, Massimiliano Mantione <massimilia...@gmail.com> wrote:

On Wed, Aug 10, 2011 at 10:51 PM, Benjamin van der Veen
<b...@bvanderveen.com> wrote:
>[...]

>> Or maybe I am not getting what "connection pooling" means in this context.
>> I think of a connection as an open and connected socket, and I don't
>> see how a connected socket can be recycled in a pool.
>
> Yeah, that's what I mean. For example, if the user makes two requests to the
> same server on one iteration of the event loop, those requests could happen
> in series over one TCP connection. If the user makes another request to the
> same server later, it would be nice to reuse that already-open connection if
> possible.

> [...]

Ok, now I think I get it: pooling is HTTP 1.1 specific, it happens
when the connection stays open and can be reused for other requests.

Maybe you could do like this... 1st of all model a sort of "HTTP end
point", which is an object that "represents the server".
You use it to make requests passing a response handler and the handler
will be called appropriately when the server answers (the usual Kayak
idiom, but as you said in reverse).

Yes, exactly. :)

Creating an "HTTP end point" can involve DNS requests, but does not
involve creating sockets.

Might be better to defer the DNS resolution until just before the socket was going to be opened?

 

Then an end point should expose an API to control how keep-alive is
handled and when an 1.1 connection will be closed (maybe with a "close
now" option, even without issuing a request).

Another thing I think about is timeouts. Should the implementation just automatically close idle connections or should it always be the user's responsibility? Hmmm. No, probably the socket implementation needs to have a timeout associated with it, because this a larger problem for all protocols…

 

The end point will of course create the socket on the fly for each
request that is issued when no socket is active.

Probably the "final" issue to fix would be how to handle multiple
requests: should the end point open more connections (sockets), or
should it serialize the requests?
Likely, also in this case it should expose an API to control its behavior.
Maybe the sensible default would be opening a new socket for 1.0
requests and queuing them on the existing socket for 1.1 requests,
unless the user asks otherwise.

Yes, 1.0 connections would have a single request only. Maybe the connection "pooling" should indeed be left to the user. The only API Kayak provides is:

ep.GetHttpConnection(c => {

    c.OnRequest(head, body, responseDelegate);

    c.Dispose();

});

In this example, Connection: close is added automatically because the connection object knows it's done for while the outgoing request message is still queued up.

On the other hand:

ep.GetHttpConnection(c => {

    c.OnRequest(head, body, responseDelegate);

});

...and later...

    c.Dispose();

There might not be any outgoing request messages in the connection's queue, in which case it would shut down the socket, and assume the server will close the connection after its response. If a response was not pending from the server, the connection is forcibly closed.

The GetHttpConnection method might also take an IConnectionDelegate to receive OnError and OnClose events.

 

Just my two cents "thinking aloud" around midnight :-)

Thanks for your thoughts. I'm also tiredly rambling on after midnight here. ;)

Cheers,

Benjamin