Benchmark unexpected results

112 views
Skip to first unread message

Stevo Novkovski

unread,
May 12, 2015, 2:31:07 PM5/12/15
to openlitespee...@googlegroups.com
I made tons of benchmark results comparing lsapi, hhvm, ols, nginx, ols cache and i received so unexpected results.
I was expecting results in this order (from fastest to slower):

1) OLS cache
2) fcgi HHVM
3) lsapi

But my results says that most nature, stable and faster is lsapi.

All tests are made on Linode 1GB/1CPU,SSD and Linode 4GB/4CPU/SSD vps.

Settings:

HHVM fcgi default settings, socket connection. version 3.6.0
OLS default settings except max connection. version 1.4.7
LSAPI php 5.6.7 with opcache default settings
LSAPI php 7-dev with opcache default settings
Nginx default settings. version 1.6.3
Load methodology: Maintain client load http://prntscr.com/74gc8k
All tests are run on HTTP 80 port.


Tests on 1GB vps:

From 50 to 1500  in 30 secounds - Wordpress + w3tc + page disk cache 

ols lsphp 5.6

ols fcgi hhvm



From 50 to 1500  in 30 secounds - hello.php (echo "Hello";)


ols lsphp 5.6


ols fcgi hhvm


From 800 to 2000 in 30 secounds - hello.php (echo "Hello";)


ols lsphp 5.6




ols fcgi hhvm

ols cache disk storage


From 800 to 2000  in 30 secounds - Wordpress + w3tc + page disk cache 


ols lsphp 5.6



ols fcgi hhvm


ols cache disk storage


ols fcgi hhvm (apc storage)


From 2000 to 3000  in 30 secounds -  hello.php (echo "Hello";)


ols lsphp 5.6




ols fcgi hhvm (cpu? http://prntscr.com/74fwuw)

ols cache disk storage

lsapi php 7



From 2000 to 3000 50x50px png image

ols

nginx 



From 1000 to 2000  in 30 secounds -  hello.php (echo "Hello";)


ols lsphp 5.6



ols fcgi hhvm


nginx fcgi hhvm

lsapi php 7




Tests on 4GB vps:


From 800 to 2000 in 30 secounds Wordpress + w3tc + page disk cache 

ols lsphp 5.6

ols fcgi hhvm



ols cache (disk storage)



ols hhvm (apc cache)


ols cache (ram disk storage)



From 2000 to 3000 in 30 secounds -  hello.php (echo "Hello";)

ols lsphp 5.6

ols hhvm (cpu? http://prntscr.com/74fqom)




ols cache (disk storage)


ols cache (ram disk storage)




Mine conclusion

  1. OLS works best with lsapi and its most natural setup on OLS.
  2. Nginx hhvm fastcgi is much faster the ols hhvm fastcgi
  3. OLS is faster and more stable in static files then nginx

Questions

  1. Do you work on lsapi hhvm and when will be ready?
  2. PHP 7 is coming this year with similar performance as hhvm and i this OLS + lsapi php 7 will beat everything.
  3. Does "max 2000 external connection limit" is just front-end limitation, since i edited ols config file and change to 4000 and everything was working fine.

George Wang

unread,
May 13, 2015, 5:53:01 PM5/13/15
to openlitespee...@googlegroups.com
Hi,

Thanks for the benchmarks!

I am surprised that nginx + hhvm is faster than ols + hhvm, we think we
have better FCGI implementation than nginx, at least for our Enterprise
product. It could be a bug somewhere. I will try it out when I get a
chance.

Cache should be the fastest, we will do some internal testing as well.

Yeah, we plan to add LSAPI interface to hhvm, just lower priority. And
LiteSpeed SAPI has been ported to PHP7, so you can give PHP7 a try see
how it stack up against hhvm.

Best regards,
George Wang

On 5/12/2015 2:31 PM, Stevo Novkovski wrote:
> I made tons of benchmark results comparing lsapi, hhvm, ols, nginx, ols
> cache and i received so unexpected results.
> I was expecting results in this order (from fastest to slower):
>
> 1) OLS cache
> 2) fcgi HHVM
> 3) lsapi
>
> But my results says that most *nature*, stable and faster is lsapi.
>
> All tests are made on Linode 1GB/1CPU,SSD and Linode 4GB/4CPU/SSD vps.
>
> Settings:
>
> HHVM fcgi default settings, socket connection. version 3.6.0
> OLS default settings except max connection. version 1.4.7
> LSAPI php 5.6.7 with opcache default settings
> Nginx default settings. version 1.6.3
> Load methodology: Maintain client load http://prntscr.com/74gc8k
> All tests are run on HTTP 80 port.
>
>
> Tests on 1GB vps:
>
> From 50 to 1500 in 30 secounds - Wordpress + w3tc + page disk cache
>
> ols lsphp 5.6
>
> <http://i.imgur.com/NOwEvp6.png>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/GWxeFcI.png>
>
>
>
> From 50 to 1500 in 30 secounds - hello.php (echo "Hello";)
>
>
> ols lsphp 5.6
>
> <http://i.imgur.com/YEAqt6X.png>
>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/bYaK47L.png>
>
>
> From 800 to 2000 in 30 secounds - hello.php (echo "Hello";)
>
>
> ols lsphp 5.6
>
> <http://i.imgur.com/BQcgP90.png>
>
>
>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/IIwr260.png>
>
> ols cache disk storage
>
> <http://i.imgur.com/iyNgJOt.png>
>
>
> From 800 to 2000 in 30 secounds - Wordpress + w3tc + page disk cache
>
>
> ols lsphp 5.6
>
> <http://i.imgur.com/YoKJp9Q.png>
>
>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/ffcyhip.png>
>
>
> ols cache disk storage
>
> <http://i.imgur.com/LA8OJUY.png>
>
>
> ols fcgi hhvm (apc storage)
>
> <http://i.imgur.com/ZcXo0ZA.png>
>
>
> From 2000 to 3000 in 30 secounds - hello.php (echo "Hello";)
>
>
> ols lsphp 5.6
>
> <http://i.imgur.com/fAZfvDc.png>
>
>
>
>
> ols fcgi hhvm (cpu? http://prntscr.com/74fwuw)
>
> <http://i.imgur.com/uZA73IZ.png>
>
> ols cache disk storage
>
> <http://i.imgur.com/x3ikLS8.png>
>
>
> From 2000 to 3000 50x50px png image
>
> ols
>
> <http://i.imgur.com/nxbUniY.png>
>
> nginx
>
> <http://i.imgur.com/xfTOax5.png>
>
>
>
> From 1000 to 2000 in 30 secounds - hello.php (echo "Hello";)
>
>
> ols lsphp 5.6
>
> <http://i.imgur.com/7hH1Spf.png>
>
>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/Fafpzo4.png>
>
>
> nginx fcgi hhvm
>
> <http://i.imgur.com/pxb4lP4.png>
>
>
> Tests on 4GB vps:
>
>
> From 800 to 2000 in 30 secounds Wordpress + w3tc + page disk cache
>
> ols lsphp 5.6
>
> <http://i.imgur.com/MSKmOxP.png>
>
> ols fcgi hhvm
>
> <http://i.imgur.com/URziKP5.png>
>
>
>
> ols cache (disk storage)
>
> <http://i.imgur.com/kNel6OU.png>
>
>
>
> ols hhvm (apc cache)
>
> <http://i.imgur.com/UYnEUMt.png>
>
>
> ols cache (ram disk storage)
>
> <http://i.imgur.com/AdEfTDx.png>
>
>
>
> From 2000 to 3000 in 30 secounds - hello.php (echo "Hello";)
>
> ols lsphp 5.6
>
> <http://i.imgur.com/CnTvTmJ.png>
>
> ols hhvm (cpu? http://prntscr.com/74fqom)
>
> <http://i.imgur.com/fMSBS6n.png>
>
>
>
>
> ols cache (disk storage)
>
> <http://i.imgur.com/2MtR5oy.png>
>
>
> ols cache (ram disk storage)
>
> <http://i.imgur.com/rPheJyc.png>
>
>
>
>
> Mine conclusion
>
> 1. OLS works best with lsapi and its most natural setup on OLS.
> 2. Nginx hhvm fastcgi is much faster the ols hhvm fastcgi
> 3. OLS is faster and more stable in static files then nginx
>
>
> Questions
>
> 1. Do you work on lsapi hhvm and when will be ready?
> 2. PHP 7 is coming this year with similar performance as hhvm and i
> this OLS + lsapi php 7 will beat everything.
> 3. Does "max 2000 external connection limit" is just front-end
> limitation, since i edited ols config file and change to 4000 and
> everything was working fine.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenLiteSpeed Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openlitespeed-deve...@googlegroups.com
> <mailto:openlitespeed-deve...@googlegroups.com>.
> To post to this group, send email to
> openlitespee...@googlegroups.com
> <mailto:openlitespee...@googlegroups.com>.
> Visit this group at
> http://groups.google.com/group/openlitespeed-development.
> For more options, visit https://groups.google.com/d/optout.

Stevo Novkovski

unread,
May 13, 2015, 7:32:46 PM5/13/15
to openlitespee...@googlegroups.com
Well i think there is no point of developing lsapi hhvm when lsapi php7 will have same or better results but its good to have it.

I edited my post and included one php7 lsapi test, performance gain is visible with single eye :)

In theory wordpress disk enhanced cache should have similar performance with ols cache because there two main function executed in pure c++

ols cache
1. Check if cache exist
2. If exist, return cached, else make html, cache it and give result.


wordpress disk enhanced 

1. Check if we have static html via rewrite rules
2. If exist, return cached, else make html, cache it and give result.

Ill update my benchmark in one month and ill just compare ols.
> To post to this group, send email to
> openlitespee...@googlegroups.com

George Wang

unread,
May 18, 2015, 5:11:53 PM5/18/15
to openlitespee...@googlegroups.com
PHP7 is about 50% faster than the others for the hello.php test. impressive!

If served from static file, should have similar result for lscache and
disk enhanced cache.

Best regards,
George Wang
> > <http://i.imgur.com/NOwEvp6.png <http://i.imgur.com/NOwEvp6.png>>
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/GWxeFcI.png <http://i.imgur.com/GWxeFcI.png>>
> >
> >
> >
> > From 50 to 1500 in 30 secounds - hello.php (echo "Hello";)
> >
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/YEAqt6X.png <http://i.imgur.com/YEAqt6X.png>>
> >
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/bYaK47L.png <http://i.imgur.com/bYaK47L.png>>
> >
> >
> > From 800 to 2000 in 30 secounds - hello.php (echo "Hello";)
> >
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/BQcgP90.png <http://i.imgur.com/BQcgP90.png>>
> >
> >
> >
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/IIwr260.png <http://i.imgur.com/IIwr260.png>>
> >
> > ols cache disk storage
> >
> > <http://i.imgur.com/iyNgJOt.png <http://i.imgur.com/iyNgJOt.png>>
> >
> >
> > From 800 to 2000 in 30 secounds - Wordpress + w3tc + page disk cache
> >
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/YoKJp9Q.png <http://i.imgur.com/YoKJp9Q.png>>
> >
> >
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/ffcyhip.png <http://i.imgur.com/ffcyhip.png>>
> >
> >
> > ols cache disk storage
> >
> > <http://i.imgur.com/LA8OJUY.png <http://i.imgur.com/LA8OJUY.png>>
> >
> >
> > ols fcgi hhvm (apc storage)
> >
> > <http://i.imgur.com/ZcXo0ZA.png <http://i.imgur.com/ZcXo0ZA.png>>
> >
> >
> > From 2000 to 3000 in 30 secounds - hello.php (echo "Hello";)
> >
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/fAZfvDc.png <http://i.imgur.com/fAZfvDc.png>>
> >
> >
> >
> >
> > ols fcgi hhvm (cpu? http://prntscr.com/74fwuw)
> >
> > <http://i.imgur.com/uZA73IZ.png <http://i.imgur.com/uZA73IZ.png>>
> >
> > ols cache disk storage
> >
> > <http://i.imgur.com/x3ikLS8.png <http://i.imgur.com/x3ikLS8.png>>
> >
> >
> > From 2000 to 3000 50x50px png image
> >
> > ols
> >
> > <http://i.imgur.com/nxbUniY.png <http://i.imgur.com/nxbUniY.png>>
> >
> > nginx
> >
> > <http://i.imgur.com/xfTOax5.png <http://i.imgur.com/xfTOax5.png>>
> >
> >
> >
> > From 1000 to 2000 in 30 secounds - hello.php (echo "Hello";)
> >
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/7hH1Spf.png <http://i.imgur.com/7hH1Spf.png>>
> >
> >
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/Fafpzo4.png <http://i.imgur.com/Fafpzo4.png>>
> >
> >
> > nginx fcgi hhvm
> >
> > <http://i.imgur.com/pxb4lP4.png <http://i.imgur.com/pxb4lP4.png>>
> >
> >
> > Tests on 4GB vps:
> >
> >
> > From 800 to 2000 in 30 secounds Wordpress + w3tc + page disk cache
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/MSKmOxP.png <http://i.imgur.com/MSKmOxP.png>>
> >
> > ols fcgi hhvm
> >
> > <http://i.imgur.com/URziKP5.png <http://i.imgur.com/URziKP5.png>>
> >
> >
> >
> > ols cache (disk storage)
> >
> > <http://i.imgur.com/kNel6OU.png <http://i.imgur.com/kNel6OU.png>>
> >
> >
> >
> > ols hhvm (apc cache)
> >
> > <http://i.imgur.com/UYnEUMt.png <http://i.imgur.com/UYnEUMt.png>>
> >
> >
> > ols cache (ram disk storage)
> >
> > <http://i.imgur.com/AdEfTDx.png <http://i.imgur.com/AdEfTDx.png>>
> >
> >
> >
> > From 2000 to 3000 in 30 secounds - hello.php (echo "Hello";)
> >
> > ols lsphp 5.6
> >
> > <http://i.imgur.com/CnTvTmJ.png <http://i.imgur.com/CnTvTmJ.png>>
> >
> > ols hhvm (cpu? http://prntscr.com/74fqom)
> >
> > <http://i.imgur.com/fMSBS6n.png <http://i.imgur.com/fMSBS6n.png>>
> >
> >
> >
> >
> > ols cache (disk storage)
> >
> > <http://i.imgur.com/2MtR5oy.png <http://i.imgur.com/2MtR5oy.png>>
> >
> >
> > ols cache (ram disk storage)
> >
> > <http://i.imgur.com/rPheJyc.png <http://i.imgur.com/rPheJyc.png>>
> >
> >
> >
> >
> > Mine conclusion
> >
> > 1. OLS works best with lsapi and its most natural setup on OLS.
> > 2. Nginx hhvm fastcgi is much faster the ols hhvm fastcgi
> > 3. OLS is faster and more stable in static files then nginx
> >
> >
> > Questions
> >
> > 1. Do you work on lsapi hhvm and when will be ready?
> > 2. PHP 7 is coming this year with similar performance as hhvm and i
> > this OLS + lsapi php 7 will beat everything.
> > 3. Does "max 2000 external connection limit" is just front-end
> > limitation, since i edited ols config file and change to 4000 and
> > everything was working fine.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "OpenLiteSpeed Development" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to openlitespeed-deve...@googlegroups.com
> <javascript:>
> > <mailto:openlitespeed-deve...@googlegroups.com
> <javascript:>>.
> > To post to this group, send email to
> > openlitespee...@googlegroups.com <javascript:>
> > <mailto:openlitespee...@googlegroups.com <javascript:>>.
> <http://groups.google.com/group/openlitespeed-development>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenLiteSpeed Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openlitespeed-deve...@googlegroups.com
> <mailto:openlitespeed-deve...@googlegroups.com>.
> To post to this group, send email to
> openlitespee...@googlegroups.com
> <mailto:openlitespee...@googlegroups.com>.

David

unread,
May 21, 2015, 4:57:52 PM5/21/15
to Stevo Novkovski, openlitespee...@googlegroups.com
Hi Stevo,

Thanks for giving us the Benchmark.
About the speed of our cache and wordpress cache, normally they are about to be the same.
I want to know how you did the test of openlitespeed with the cache module because sometimes cache may be disabled if not correctly configured. And also do you get any other results?

Thanks again.
David



On 5/13/2015 7:32 PM, Stevo Novkovski wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.
To post to this group, send email to openlitespee...@googlegroups.com.

Stevo Novkovski

unread,
May 23, 2015, 8:54:49 AM5/23/15
to openlitespee...@googlegroups.com, novkovs...@gmail.com
I just enabled cache for every php file and of course response had "x-litespeedcache: public" header.

I think osl cache can never be faster then wordpress w3tc simple becouse:

1. No matter how complex are  w3tc settings for page cache, every rule is generated in rewrite, so web server just parse the rewrite and if there is file, simple just give that file as result.
2. Ols cache do the same + for evey request it checks if cache is expired, stale, first checks for private then for public cache etc....

But to be honest, OLS cache is made to be generic caching module.

Reply all
Reply to author
Forward
0 new messages