TechEmpower Round 4

[James L.]

Let keep all Play related framework discussion on Framework Benchmark in this thread.

[Brian Hauer]

Thanks for the thread, James!  We will be posting Round 4 tomorrow (May 2).

Round 4 includes Play-scala with MongoDB, the Play-scala implementation of our new Fortunes test, and resolves the errors with Play1 from the previous round.

However, we have not yet incorporated improvements that I have heard are in the works for Play.  I hope those will be in for Round 5.

I just posted some other details at https://groups.google.com/d/msg/framework-benchmarks/lW1hyXw4FG4/cRlQYiogcNQJ

[Brian Hauer]

Hi all,

We've posted Round 4!  Here are the URLs.  The results are here:

http://www.techempower.com/benchmarks/#section=data-r4

And our blog has some observations and notes:

http://www.techempower.com/blog/2013/05/02/frameworks-round-4/

As always, we look forward to your feedback.  Questions, suggestions, critiques, or anything else.  Thanks again to everyone who has helped and contributed!

[Tom Carchrae]

I don't understand how/why play1 can be so much slower the play1-sienna.  I'll run some local tests and compare them.  Does running inside resin really make it that much faster?.... Hmm.

Tom

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Grant Beaty]

Why isn't Lift listed as full-stack? Its got more db connectivity options, form-handling logic, etc. than play2. Its a fuller-stack in every respect I can think of.

Would you guys accept a submission with play2 using the Activate ORM, or are you only using the framework's built-in ORMs?

My $0.02 on play2 performance: If I was a play2 dev, I would be looking very hard at spray.io and Akka IO, and not jumping to improve performance at the network level. I haven't done much with Netty, but IMO it shows its age (perhaps venerably) compared to newer more functional approaches. Play2 is still faster than most full-stack frameworks, and is not an order of magnitude slower like Rails and Django are. Gemini I just don't know what to make of ;)

/Grant

[Adis]

Gemini information: https://groups.google.com/forum/?fromgroups=&hl=en-GB#!topic/framework-benchmarks/p3PbUTg-Ibk

[Brian Hauer]

Oops!  Thanks for catching the mis-characterization of Lift.  I've just fixed that up.

Our basic thinking on additional test permutations is this: if there is widespread or official word from the framework's community about the best practices, we'd prefer to use that singular test.  If however, there is debate or uncertainty about the best practices, it may be preferable to include multiple permutations.  For example, a previous contributor provided both "play1" and "play1-siena."

So I'd simply ask that you use your best judgment in determining whether a Play test with Activate would be a useful addition.  If you submit the pull request, I believe we'll know how you feel about its value, so we'll accept it.

I recommend omitting Gemini in comparisons.  That is our internal closed-source framework that we've included for our own consumption.  Yes, we're happy with its performance and we absolutely wanted the data, but we have the unique luxury of not having to support an open source community using it.  The project is not about Gemini and I am in here because I want Play to look good in key performance metrics.

My strong preference would be that the open source frameworks see the project as an opportunity to improve in a friendly competition.  Incidentally, a future version of this results web site will allow you to hyperlink directly to a specific comparison view.  For example, one could narrow in on full-stack only and drop Gemini from the charts and then share that for a conversation.

[Andy]

On Friday, May 3, 2013 10:45:21 AM UTC-4, Brian Hauer wrote:

I recommend omitting Gemini in comparisons.  That is our internal closed-source framework that we've included for our own consumption.  

Any chance you can open source Gemini? Would like to see what gives it such great performance. 

[Brian Hauer]

We are considering doing that, in part because it would be fair to the other frameworks in our benchmarks project.  We don't think we're doing anything particularly remarkable in Gemini, but perhaps others could pick and choose things they like and use them elsewhere.  The documentation and our ability to support its usage would be very limited.  See my earlier note on the matter here: https://groups.google.com/forum/#!topic/framework-benchmarks/p3PbUTg-Ibk

That said, let's keep this thread about Play.

[Rémy Darcel]

The results seems to be very strange at first comparing to my own tests when benchmarking servlet versus play with a simple Tomcat so I have made a simple test with Resin and play 1.2.5 on my laptop (core i5, 4GB ram, Windows 7, SSD hardrive, a lot of software running at the same time) :

• Deploy a simple hello world application as a war on resin
• Start the same simple hello world application using standard play deployment 
• Use ab to realize the benchmark : ./ab.exe -n 100000 -c 60
  

And the results where very surprising : Play was runing much faster on resin than using netty : a~5000 req/s compared to ~1500 req/s.

I admit that it is absolutely not scientifical but maybe on one the reason why servlet based frameworks are such faster in the techempower benchmark is that Resin that seems to be very optimized.

[Brian Hauer]

Hi,

Resin is indeed a well-tuned Java Servlet container.  We selected it for these tests because (a) we've used it extensively elsewhere and (b) we initially also included Tomcat in Round 1 for Servlet tests but dropped Tomcat because it was consistently somewhat slower than Resin.  For Round 1 we didn't want the additional noise in the results charts.

However, since then, especially now that we have a filtering user interface for the results, we are considering re-introducing Tomcat (or perhaps Wildfly when it is released) as another test dimension for Servlet-based frameworks.

That said, the differential you are seeing between Play with Resin versus Play with Netty is considerably more dramatic than the differential between Tomcat and Resin in our tests for Servlet-based frameworks.  If I recall correctly, the Tomcat versus Resin differential for our tests was in the 10% to 15% range.

Also, if possible, I encourage you to use a multi-threaded load tool rather than ab (ApacheBench).  We use Wrk, but there is also WeigHTTP, which is a good multi-threaded clone ot ApacheBench.  I believe WeigHTTP is available on Windows.

A multi-threaded tool is not likely to invert the results you're seeing, but generally speaking will reveal a more realistic measurement of the server's true top-end capacity.  ApacheBench can fail to sufficiently exercise a high-performance server.

I assume the Resin versus Netty differential converges if you switch to one of the more computationally intensive tests such as the multiple-query test or the new Fortunes test (queries, collections, sorting, and server-side templates)?  Are you able to run those?  If not, let me know how we can help.

[Rémy Darcel]

Hello,

Yes I admit that the value of the test is quite limited but the result was very surprising.

I will adapt the play-scala bench with play2war in order to to realize the test on a linux vm to have a more precise indication on the differences between netty/resin in the case of the play framework and publish the result.

[Manuel Bernhardt]

Hi,

just a suggestion for the TechEmpower benchmarks if I may: I'd run
them on the Oracle JDK for Java frameworks, just to see what would
happen.

I would not be surprised if this would shuffle things around quite a bit.

Manuel

[Brian Hauer]

Hi Manuel,

We've not used the Oracle JDK in any of our published results, but we've spot checked with the Oracle JDK a few times in the past.  Nothing caught our attention when comparing the two.  Of course, it has been a while since we did that, so I agree there may be some interesting results.

Thanks for the note!

[Brian Hauer]

Minor update: I've just posted a revision to the results web site that captures filters to the URL.

The following is a view of Round 4's data limited to just the open source full-stack frameworks, which is a better way to compare Play.

http://www.techempower.com/benchmarks/#section=data-r4&c=1&f=zik0zi-13ydj3

More detail: https://groups.google.com/d/msg/framework-benchmarks/d-RsysmdSmg/wzyDVx2qWEwJ

[Chanan Braunstein]

The URL you posted omits the name of the framework on the graph (At least on chrome on win 7) - it only shows colors now.

[Brian Hauer]

Oops!  Haha.  Thanks for pointing that out.  :)

Chrome doesn't support the zero-parameter element.cloneNode method of course.  Give it another go.

[Damien Lecan]

Hello,

As Play2War author, I'm really interested in your bechmarks on Resin :)
However, please notice that Play2war has never been tested with Resin ...

Damien

2013/5/8 Rémy Darcel <rda...@gmail.com>:

[Rémy Darcel]

Hello,

I have just realized the installation of the benchmark on EC2 so I Think I will be able to post the results in the next days.

[Damien Lecan]

Hi Rémy,

Any results ?

Damien

2013/5/14 Rémy Darcel <rda...@gmail.com>: