HttpClient MAX_CONNECTIONS_PER_ROUTE

[tim_darby]

Hi,

I've been debugging into a performance issue with my use of SimpleDB,
and I seem to have (possibly) hit a problem with the number of
concurrent connections typica is prepared to use when talking to
SimpleDB.

I have multiple threads in a service (ThreadPoolExecutor based), which
didn't appear to be parallelising well enough. When I dug into this
some more, it seemed that HttpClient is throttling my use of http
connections to the SDB endpoint as whilst "http.conn-manager.max-
total" parameter on the connection manager is being set to 100,
"http.conn-manager.max-per-route" is not being set (see below snippet
from AWSQueryConnection.java from 1.7).

The "//" does seem to be the harbinger of doom as without it, from
what I can tell of HttpClient 4, I wouldn't be limited to the default
of 2 connections per destination. Trouble is, typica doesn't seem to
allow me to monkey with the params of the HttpClient (setHttpClient
but not getHttpClient, and setting up a scratch version feels like a
cludge too far).


params.setParameter(AllClientPNames.MAX_TOTAL_CONNECTIONS, new
Integer(maxConnections));
params.setParameter(AllClientPNames.VIRTUAL_HOST, getServer());
// params.setParameter(AllClientPNames.MAX_CONNECTIONS_PER_ROUTE, new
ConnPerRouteBean(maxConnections));

Does anyone else have any knowledge they can shed on this - I've
certainly seen people making successful use of typica to higher
concurrencies, but I can't from here see how I can do it without being
a bad lad and creating many Domain instances, which is to miss the
point rather (and create probably a lot of cleanup threads).

Tim

[tim_darby]

At the risk of appearing to engage in the slightly mad practice of
answering myself...

... I just tried out running a locally built typica with that line
commented back in, and it certainly does appear to reinvigorate my
parallellism of requests, with a function which was taking ~30 seconds
throttled at 2 concurrent connections taking now just 10 seconds with
the MAX_CONNECTIONS_PER_ROUTE re-enabled to be the same as
MAX_TOTAL_CONNECTIONS.

Any chance that line could be reinstated?

Tim

p.s. For completeness I just checked using svn blame and it seems to
have been part of the general switch to HttpClient 4, so perhaps was
commented out whilst getting v4 working, but accidentally not
reinstated?

[David Kavanagh]

I seem to recall that it wasn't available in HttpClient 4. Let me check again though.

--
You received this message because you are subscribed to the Google Groups "typica" group.
To post to this group, send email to typ...@googlegroups.com.
To unsubscribe from this group, send email to typica+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/typica?hl=en.

[David Kavanagh]

OK, that's interesting. Do you set it to an Integer? It seems to want this; http://hc.apache.org/httpcomponents-client/httpclient/apidocs/org/apache/http/conn/params/ConnPerRouteBean.html

[tim_darby]

I just uncommented what was already there, but commented out :-

params.setParameter(AllClientPNames.MAX_CONNECTIONS_PER_ROUTE, new
ConnPerRouteBean(maxConnections));

Which is just setting the default max connections for routes not
explicitly defined, which seems logical enough for the
AWSQueryConnection use case as any given instance will 99% certain
only ever get used for one host anyway, which keeps things simple ...
explicitly defining the max connections for a particular route... now
that'd be actual work!

On Jul 26, 9:41 pm, David Kavanagh <dkavan...@gmail.com> wrote:
> OK, that's interesting. Do you set it to an Integer? It seems to want this;http://hc.apache.org/httpcomponents-client/httpclient/apidocs/org/apa...

>
>
>
> On Mon, Jul 26, 2010 at 4:37 PM, David Kavanagh <dkavan...@gmail.com> wrote:
> > I seem to recall that it wasn't available in HttpClient 4. Let me check
> > again though.
>

> > On Mon, Jul 26, 2010 at 4:33 PM, tim_darby <tim_da...@scee.net> wrote:
>
> >> At the risk of appearing to engage in the slightly mad practice of
> >> answering myself...
>
> >> ... I just tried out running a locally built typica with that line
> >> commented back in, and it certainly does appear to reinvigorate my
> >> parallellism of requests, with a function which was taking ~30 seconds
> >> throttled at 2 concurrent connections taking now just 10 seconds with
> >> the MAX_CONNECTIONS_PER_ROUTE re-enabled to be the same as
> >> MAX_TOTAL_CONNECTIONS.
>
> >> Any chance that line could be reinstated?
>
> >> Tim
>
> >> p.s. For completeness I just checked using svn blame and it seems to
> >> have been part of the general switch to HttpClient 4, so perhaps was
> >> commented out whilst getting v4 working, but accidentally not
> >> reinstated?
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "typica" group.
> >> To post to this group, send email to typ...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> typica+un...@googlegroups.com<typica%2Bunsu...@googlegroups.com>
> >> .