Maximize LWP https performance
Daniel Parry
I am attempting to load test an https web application using a
custom perl application. The order of requests made to the web
application is important so I can't just fire off all the requests
in parallel a la LWP::Parallel. So, my latest approach has been to
fork multiple times to achieve parallelism, with each fork
simulating a web application user. Via this technique I achieve a
sustained rate of about 12 requests / second which I feel, perhaps
incorrectly(?), that I should be able to improve upon.
I notice that some of the requests take an unusually long time to
complete. Seemingly unrelated to the web application server
responsiveness. Thus, I have two theories:
1) Contention over use of network. Requests are getting stacked up
at the mercy of the network scheduler.
I'm not entirely sure of a good way of confirming or disproving
this? It may be that splitting the execution across multiple hosts
is a way forward but I'd like to maximize performance on a single
host first before duplicating it across additional hosts...
2) SSL connection is biting.
Profiling a single process (no forking) suggests to me that a fair
bit of time is spent dealing with SSL connections.
%Time ExclSec CumulS #Calls sec/call Csec/c Name
78.0 1.970 1.970 180 0.0109 0.0109 Crypt::SSLeay::Conn::connect
8.16 0.206 0.206 114428 0.0000 0.0000 HTTP::Headers::_header
6.81 0.172 0.358 15674 0.0000 0.0000 Net::SSL::read
5.31 0.134 0.134 15674 0.0000 0.0000 Crypt::SSLeay::Conn::read
4.32 0.109 0.143 1854 0.0001 0.0001 HTML::Parser::parse
I am using keep alives:
my $userAgent = LWP::UserAgent->new( keep_alive=>1 );
So... I was wondering if anyone has done heavy load testing on
https sites via perl previously and could give me a few pointers
or hints please!
In fact, any help would be much appreciated!
Best wishes,
Daniel
xho...@gmail.com
> Hello,
>
> I am attempting to load test an https web application using a
> custom perl application. The order of requests made to the web
> application is important so I can't just fire off all the requests
> in parallel a la LWP::Parallel. So, my latest approach has been to
> fork multiple times to achieve parallelism, with each fork
> simulating a web application user. Via this technique I achieve a
> sustained rate of about 12 requests / second which I feel, perhaps
> incorrectly(?), that I should be able to improve upon.
>
> I notice that some of the requests take an unusually long time to
> complete. Seemingly unrelated to the web application server
> responsiveness.
What is the "seemingly unrelated" conclusion based on?
> Thus, I have two theories:
>
> 1) Contention over use of network. Requests are getting stacked up
> at the mercy of the network scheduler.
>
> I'm not entirely sure of a good way of confirming or disproving
> this?
Make a dummy web application that does nothing but serve static files.
For example, go through the web app, saving the response for each request.
Then copy these responses to web server and make your client requests these
static files rather than doing actual dynamic requests. If your network is
the problem (unlikely, I suspect), you will not see much improvement. If
it isn't the problem, you should see a big improvement.
> It may be that splitting the execution across multiple hosts
> is a way forward but I'd like to maximize performance on a single
> host first before duplicating it across additional hosts...
>
> 2) SSL connection is biting.
Change your app to allow non-SSL connections, then run the benchmark again
that way and see if it speeds up.
>
> Profiling a single process (no forking) suggests to me that a fair
> bit of time is spent dealing with SSL connections.
You seem to be profiling the client program, not the web-app server
programs, right?
>
> %Time ExclSec CumulS #Calls sec/call Csec/c Name
> 78.0 1.970 1.970 180 0.0109 0.0109 Crypt::SSLeay::Conn::connect
> 8.16 0.206 0.206 114428 0.0000 0.0000 HTTP::Headers::_header
> 6.81 0.172 0.358 15674 0.0000 0.0000 Net::SSL::read
> 5.31 0.134 0.134 15674 0.0000 0.0000 Crypt::SSLeay::Conn::read
> 4.32 0.109 0.143 1854 0.0001 0.0001 HTML::Parser::parse
Was this generated with dprofpp, dprofpp -r, or dprofpp -u, or something
else?
What was the header from the profile?, e.g.
Total Elapsed Time = 1.67 Seconds
User Time = 0.61 Seconds
Xho
--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.