Why is Play! so low on these tests?

[Ittay Dror]

http://www.techempower.com/blog/2013/03/28/framework-benchmarks/

(While this may seem relevant to the Play! mailing list, I thought it would be interesting to the folks here because Play! relies on Scala and Akka to do  its stuff)

(Also: I'm not trying to start a troll here)

Play! ranks well below other Java frameworks. I expected higher results being that it is not based on a scripted language, is full stack and uses evented I/O using Akka & Netty (Netty standalone ranks higher in the benchmark)

I'm sure all disclaimers about such benchmarks apply here, and that maybe this isn't the best place for Scala/Akka to shine since the tests seem to be low on internal business logic (just fetching records from a DB and serializing to json). But services that fetch data from a DB and send it over json are probably a large share of the target market for Scala/Akka.

Regards,
Ittay

[S Ahmed]

Depends on the scala code that's written i.e. http://www.youtube.com/watch?feature=player_embedded&v=IfLwuCVRQTU

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[yossarian]

The benchmark code is java and looks like it's performing blocking i/o:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java

See Sadache's material:

http://sadache.tumblr.com/post/42351000773/async-reactive-nonblocking-threads-futures-executioncont

[Nils Kilden-Pedersen]

On Sun, Mar 31, 2013 at 10:27 AM, yossarian <mark....@yahoo.com> wrote:

The benchmark code is java and looks like it's performing blocking i/o:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java

I didn't look at the code, but that should have nothing to do with anything. Non-blocking, if anything, is slower, but allows greater scale.
 

See Sadache's material:

http://sadache.tumblr.com/post/42351000773/async-reactive-nonblocking-threads-futures-executioncont

[Fernando Racca]

There's a scala version, but somebody has to run it and update the charts

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-scala/app/controllers/Application.scala

[yossarian]

It's been updated to use non blocking i/o...

[Fernando Racca]

After round 2, the performance is clearly better, but still far away from the likes of netty, vertx, wicket and tapestry.

Any suggestions why this is the case? 

I would expect Play-scala to be on par with netty.

Fernando

[Johannes Rudolph]

On Mon, Apr 8, 2013 at 11:19 AM, Fernando Racca <fra...@gmail.com> wrote:
> I would expect Play-scala to be on par with netty.

Nothing can be on par with netty for such a simple benchmark. That's
because play is currently based on netty and just adds stuff. You
can't get faster this way.

See how easily play beats netty in the database tests (because they
didn't even implement them for netty). A more useful comparison would
more prominently focus on things as:

* scalability (concurrency of 256? Why not 10000?)
* programming model
* integration
* tooling
* ...

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Kevin Wright]

Part of the issue is that he's not testing what he thinks he's testing.  A good portion of the timing data is going to be all about JSON serialisation/deserialisation.

As for the speed of the servlet and netty tests? The telltale lines are here:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/servlet/src/main/java/hello/JsonServlet.java#L8

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/netty/src/main/java/hello/HelloServerHandler.java#L29

Being async has nothing to do with with, we're just seeing the (already well-known) fact that jacskon is pretty darned quick.

I have every confidence in the ability of macros to generate a highly optimised parser, pulling as much of the cost as possible into compile-time.  We'll turn the result around yet, just as soon as someone implements that macro-based JSON library. (because Scala really doesn't yet have enough JSON libraries…)

Jackon still has the advantage that it can make use of static methods, as much beloved of highly optimised Java code - but we're working on levelling that playing field as well :)

--
Kevin Wright
mail: kevin....@scalatechnology.com
gtalk / msn : kev.lee...@gmail.com

quora: http://www.quora.com/Kevin-Wright

google+: http://gplus.to/thecoda

twitter: @thecoda

vibe / skype: kev.lee.wright

steam: kev_lee_wright

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

[Pascal Voitot Dev]

On Mon, Apr 8, 2013 at 11:39 AM, Kevin Wright <kev.lee...@gmail.com> wrote:

Part of the issue is that he's not testing what he thinks he's testing.  A good portion of the timing data is going to be all about JSON serialisation/deserialisation.

As for the speed of the servlet and netty tests? The telltale lines are here:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/servlet/src/main/java/hello/JsonServlet.java#L8

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/netty/src/main/java/hello/HelloServerHandler.java#L29

Being async has nothing to do with with, we're just seeing the (already well-known) fact that jacskon is pretty darned quick.

I have every confidence in the ability of macros to generate a highly optimised parser, pulling as much of the cost as possible into compile-time.  We'll turn the result around yet, just as soon as someone implements that macro-based JSON library. (because Scala really doesn't yet have enough JSON libraries…)

I'm not sure to understand how you could improve string to json parsing using macro ? (BTW, I'm play-json contributor)

Thanks

Pascal

[Kevin Wright]

"Nothing can be on par with netty for such a simple benchmark. That's
because play is currently based on netty and just adds stuff. You
can't get faster this way."

Enough of the crazy talk!  Just try sticking an instance of Varnish in front of the thing, or mayby even use Amazon's own cloud front service… Then you'll see the effects that "adding stuff" can have on speed :)

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Kevin Wright]

If you know for certain that you're going to be serialising to/from a pre-determined structure, then it would be possible to implement this as a state machine, as generated at compile time.  This is similar to the approach that regexes take.

For that matter, you could have a macro actually implement the deserialiser by simply generating one large regex for the entire json structure.  Which would double the benefit if we were also performing regex compilation at compile time.

Done this way, there's also no need to perform a two-phase (de)serialisation that first goes through an intermediate DOM.

[Pascal Voitot Dev]

interesting approach ;)

I'll think about it!

[Kevin Wright]

Doesn't even need macros, to be honest.  Grab the structure of a case class using any of the pre-existing techniques, then generate and compile your regex up front.  Either that, or make it a lazy val.

[Josh Suereth]

There's actually some, quite awesome, research at EPFL on how to efficiently use macros for generic serialization.   IIRC, the binary ssrialization speeds they're seeing are quite impressive vs. Leading java binary serializers.

Would love to see a benchmark over json and Jackson.

[Kevin Wright]

Ooooh, you've got me all curious now.  Does anybody have anything up on GitHub yet?  I'm itching to have a play.

[Josh Suereth]

NO, but I decided to become an advocate :).  It's a branch of Scala, so it needs some Reflection/Macro fixes to make it into trunk before it's generally usable by the "masses", or before it could even be a separate library.

You can also see that the new scala-reactive design made it into ECOOP: http://info-web.lirmm.fr/ecoop13/index.php/accepted-papers

[Johannes Rudolph]

On Mon, Apr 8, 2013 at 11:39 AM, Kevin Wright <kev.lee...@gmail.com> wrote:

> Part of the issue is that he's not testing what he thinks he's testing. A
> good portion of the timing data is going to be all about JSON
> serialisation/deserialisation.

I doubt that this little bit of json serialisation is more important
than any of HTTP protocol serialisation/deserialisation that has to be
done.

Here's a profile captured during the run of the netty benchmark from
the link above.

https://gist.github.com/jrudolph/5336279