Round 9 of TechEmpower benchmark results have finally been posted

[Web Dev]

Looks like they included a 10-gigabit environment to further test the scalability of the frameworks.

http://www.techempower.com/benchmarks/

[Tim Fox]

On 02/05/14 03:39, Web Dev wrote:

Looks like they included a 10-gigabit environment to further test the scalability of the frameworks.

Yep. Unfortunately these latest tests were actually run using a config which was frozen some weeks/months back and use Vert.x 2.1M3 (which was released in January), so we don't see the benefits of the performance tuning that we've done since then (both in Vert.x and Netty). But hopefully in the next round we should see some significant improvements especially in the peak results.

http://www.techempower.com/benchmarks/

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rick Hight]

Round 9!


http://www.techempower.com/blog/2014/05/01/framework-benchmarks-round-9/

http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=plaintext

2,263,618 requests per second! With no errors! WOW! I count that as 3rd. Or 5th if you don't mind a few dropped connections. http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=plaintext Hot damn!

JSON serialization

http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=json

Vertx is at Jackson 2.2. If you bumped up to 2.3 or 2.4 rc you would probably have come in a few places higher. It is tight and jackson improved enough to give you a 1/5% to 1% or so.

(I have played around quite a bit with Jackson JSON benchmarks. Just a little.)

When is the next one of this coming out?

I want to play.









779,112

[Tim Fox]

This was using an old version (see my last post)

--

[Rick Hightower]

You wrote it while I was still writing mine, and chatting with three people about the benchmark online.

I think the most interesting thing about this benchmark (I have been following it since the first one). Is the frameworks that have dropped out.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/uaHdkQHZJMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Rick Hightower
(415) 968-9037
Profile 

[Tim Yates]

vertx seems to do better on i7 hardware vs EC2 (at least in the rankings)

[Tim Fox]

On 02/05/14 10:10, Rick Hightower wrote:

You wrote it while I was still writing mine, and chatting with three people about the benchmark online.

I think the most interesting thing about this benchmark (I have been following it since the first one). Is the frameworks that have dropped out.

Interesting. Spray and Play don't seem to be there any more. Any others?

[Rick Hightower]

Spring is gone.

[Rick Hightower]

I might be crazy but...

if vertx can do 600K with plain text and only 200K with JSON serializing then the serializing JSON is taking up to 400K overhead. (i7 JSON)

So really at that point, you are more benchmarking Jackson than Vertx.

And if they are going to do that.. then... Hmm.....

Well then you would want a very small and focused serializer that was optimized to run in an event loop style programming model.

That is a lot of overhead to play with. A 10% serialization gain equate to a 3% gain when most of the competitors are just a few points apart of the JSON serialization bit.

I think we can do a lot better than 10%.

[Rick Hightower]

Also all of the winners for single query are using MySQL or PostgreSQL.

Let's move Single Query over to MySQL. :)

[Rick Hightower]

Ok.. last one then I am going to bed...

Vertx is what?

Say it with me...

Multi-lingual!

Can't we enter in Vertx 4 to 6 times for each of these?

Vertx. jython

Verrx java

Vertx Groovy

Vertx Scala
Vertx JS

I mean you know what is more fun then whipping Node.js?

It would be fun to beat grails at groovy and scala at the 57 frameworks that entered in this benchmark.

Whipping node.js at JS! I mean I would got to SF wearing a sandwich board with those graphs.

BTW rinjojs beats node.js for single query, and rinjo.js whips their ass for mutliquery (ours too).

Also....

Vertx. jython mysql
Verrx java  mysql
Vertx Groovy mysql
Vertx Scala mysql
Vertx JS mysql

--Toublemaker

aka Rick

[Tim Fox]

Agreed

[Rick Hightower]

Agreed?

So vague... That I am a troublemaker. :)

[Tim Fox]

lol, agreed we should submit more entries. Trouble is, all of this takes time, so less time to spend on other things.

Btw... I have asked for our round 9 results to be withdrawn. Same reasoning as Spray guys https://github.com/TechEmpower/FrameworkBenchmarks/pull/842

[Rick Hight]

I will sign up for helping with groovy. and running the JSON tests with Boon.

I will also look into switching single query to MySQL. 

When is the next one?

Who on this list can take Scala?

Who can take JS?

...

[Arno Schulz]

Stupid question, but are there internal tools to measure metrics like these in Vertx (I have used the HTTP test, and I saw that Oliver had an EB test for the 1.3 branch)?

Perhaps this could eventually be a standalone project on GitHub that you clone and run and test against vertx with various changes (like switching to boon).

[Olaf Geibig]

I was quite confused that vert.x didn't show up in the top 10. Just now when I just advertised vert.x as the framework I'd like to use for the new utility project for my client. I mean old version or not vert.x would be in the top 10, I guess even top 5, anyway. The good ranking at TechEmpower is a good argument to 'sell' vert.x because it simply saves money, especially in the cloud. But now I got the laughs that it didn't even complete the benchmark and thus must be not mature enough for business. Well, I understand your reasons but that vert.x is now listed as unable to pass the benchmark is quite misleading. Anyway, I guess I will have more chances to introduce vert.x to a client in the future.

[Tim Fox]

Hi Olaf-

Vert.x did complete the benchmarks and did very well, near the top of
the table. However, I withdrew the results because the version that was
used in the benchmarks was pretty old and there had been several
performance optimisations since then so the Vert.x results should have
been even better. We'll be back in for round 10, assuming there is not
such a big delay between code freeze for the publishing of the
benchmarks and release.

[Rick Hightower]

When is round 10?

I want to make sure I plan ahead. 

I have sharpened my tools. 

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/uaHdkQHZJMc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[El Gazali]

In case you still wanted to see or have an idea on how vert.x did in round 9 with that old version.

http://www.infoq.com/news/2014/05/benchmark-web-framework

El

[JC]

Am Freitag, 2. Mai 2014 11:21:35 UTC+2 schrieb Rick Hight:

Spring is gone.

 
What they have/had under the name "Spring" is/was a TechEmpower inhouse development, nothing to do with SpringSource.

Anyway, it's a bit frustrating to hear after each round that either the MongoDB was outdated or the optimizations were out or whatever else so the classification shoud have been even better if only... I'm thinking, maybe it happens the same way with other frameworks (tested versions missing some optimizations) so after all the classification could be realistic...

S

[Norbert Sándor]

I've just run into this issue.
Round 11 has already been published and vert.x is not on the list...
See the page: https://www.techempower.com/benchmarks/

Any news on including vertx in the list?

Thanks,
Norbi

[Paulo Lopes]

Hi Norbert,

There are 2 implementations on techempower github reppo:

* vertx
* vertx-web

I think once there is a new round you'll get results, as you can see from the techempower site they run the benchmark like twice per year but i've no idea when they will run it again...

[Norbert Sándor]

Nice, thanks! :)

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/uaHdkQHZJMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/bb9636fb-3026-44aa-b351-6471903f3152%40googlegroups.com.