Perlbal patch with remove fields option

[Nicolas Rochelemagne]

Hi there, I've worked during the past days on benchmarking perlbal as
a server and as reverse proxy...
I monitored memory usage, cpu... number of request per seconds on
several conditions. ( I noticed that load balancing on pool containing
a dead node gives bad result, I will have a look later if this part
can be improve )

I also used NYTProf to see where time was spent. Of course ( removing
the waiting time ) the main time is spent in external module such as
Danga::Service or IO::Handle.
But I noticed something quite interesting, many time is lost in the
Hash::Util::lock_ref_keys sub. This function is used by fields::new
since perl 5.009.

I think that using fields is a really good technique during
development stage, and code should not be changed. But once we would
like to set the server as a production one, it can be a good idea to
disable this part.
So I've worked on a ( very ) simple patch that could allow any user to
disable field::new usage by setting PERLBAL_REMOVE_FIELDS to 1 as an
env var. ( In this case only field::new is mocked ).
What do you think about this idea ?

Benchmark I've done so far show that this simple patch can improve
performance by 30 %.
You could access to the patch there : https://rt.cpan.org/Public/Bug/Display.html?id=72671

I used ApacheBench ( known as ab ) to generate 100,000 requests on a
perlbal reverse proxy set with 2 nodes ( from a distinct server )
I launched 5 times ab generating 100,000 requests with concurrency =
100 ( ~ 140 sec / test ). I do not restart server between two tests,
this explain why number is decreasing from one test to another one.
> ab -n 100000 -c 100 http://127.0.0.1/

Here are the results of my bench :

## remove_fields = 0
1 - 765.09 req/sec
2 - 726.90 req/sec
3 - 718.17 req/sec
4 - 711.89 req/sec
5 - 706.41 req/sec

average = 725 req / sec

## remove_fields = 1 [ PERLBAL_REMOVE_FIELDS=1 ]
1 - 988.02 req/sec
2 - 948.28 req/sec
3 - 930.01 req/sec
4 - 921.05 req/sec
5 - 917.66 req/sec

average = 941 req / sec

=> Delta = + 29.6 %

Any comment will be appreciate
thanks for reading
___________________________________
Nicolas Rochelemagne

[Nicolas Rochelemagne]

thanks for reading

___________________________________
Nicolas Rochelemagne

rochel...@cpanel.net

(713) 529-0800 x-3077

cPanel, Inc

[dormando]

> Hi there, I've worked during the past days on benchmarking perlbal as
> a server and as reverse proxy...
> I monitored memory usage, cpu... number of request per seconds on
> several conditions. ( I noticed that load balancing on pool containing
> a dead node gives bad result, I will have a look later if this part
> can be improve )

"bad result"?

> I also used NYTProf to see where time was spent. Of course ( removing
> the waiting time ) the main time is spent in external module such as
> Danga::Service or IO::Handle.
> But I noticed something quite interesting, many time is lost in the
> Hash::Util::lock_ref_keys sub. This function is used by fields::new
> since perl 5.009.

Neat!

> I think that using fields is a really good technique during
> development stage, and code should not be changed. But once we would
> like to set the server as a production one, it can be a good idea to
> disable this part.
> So I've worked on a ( very ) simple patch that could allow any user to
> disable field::new usage by setting PERLBAL_REMOVE_FIELDS to 1 as an
> env var. ( In this case only field::new is mocked ).
> What do you think about this idea ?

I have no idea! I'd need to look at it more closely. Anyone else on the
list understand this better than I do?

> Benchmark I've done so far show that this simple patch can improve
> performance by 30 %.
> You could access to the patch there : https://rt.cpan.org/Public/Bug/Display.html?id=72671

I do have some patch feedback though:

- You did a lot of fiddling with MANIFEST/etc. Can you split all of that
into a distinct patch with some commentary on why you made those changes?

- Can you clean up the style of the new .pm files to match the rest of the
project a bit more? Headers, spacing, etc. I'm trying to add less code
with distinct style code :P

> I used ApacheBench ( known as ab ) to generate 100,000 requests on a
> perlbal reverse proxy set with 2 nodes ( from a distinct server )
> I launched 5 times ab generating 100,000 requests with concurrency =
> 100 ( ~ 140 sec / test ). I do not restart server between two tests,
> this explain why number is decreasing from one test to another one.
> > ab -n 100000 -c 100 http://127.0.0.1/
>
> Here are the results of my bench :
>
> ## remove_fields = 0
> 1 - 765.09 req/sec
> 2 - 726.90 req/sec
> 3 - 718.17 req/sec
> 4 - 711.89 req/sec
> 5 - 706.41 req/sec
>
> average = 725 req / sec
>
> ## remove_fields = 1 [ PERLBAL_REMOVE_FIELDS=1 ]
> 1 - 988.02 req/sec
> 2 - 948.28 req/sec
> 3 - 930.01 req/sec
> 4 - 921.05 req/sec
> 5 - 917.66 req/sec
>
> average = 941 req / sec
>
> => Delta = + 29.6 %

Excellent! 30% isn't too shabby for a first shot. Curious though; were you
running these tests with XS::HTTPHeaders enabled? If not, could you re-run
the tests with it enabled? The performance improvement may be even higher.

[Brad Fitzpatrick]

I remember back in the day when "use fields" made things faster...

[Nicolas Rochelemagne]

Thanks for your quick answers. All the tests performed bellow have been done with XS::HTTPHeaders enabled.

I think that "use fields", is still faster than any other object flavor ( Moo, Mo, M... ), but the lock function slow the whole process... and as for answering to one request we need to create many objects, a lot of time is lost there.

From my point of view it's a very good thing to keep using it, as it avoid many mistake by checking keys during the compilation process, but it can be helpful to disable it in production environment.

I notice that my patch was not so easy to merge, so I cherry-pick it and push it on a github PR.

I will clean up the style.

___________________________________
Nicolas Rochelemagne, Developer

rochel...@cpanel.net

(713) 529-0800 x-3077

cPanel, Inc