TechEmpower/FrameworkBenchmarks

[Simon Oberhammer]

You've probably seen this: http://www.techempower.com/blog/2013/04/05/frameworks-round-2/

I'm working on getting Ringo in one of the next rounds:

https://github.com/oberhamsi/FrameworkBenchmarks/tree/master/ringojs

I did test with apache bench on my machine and I would expect us to perform at about the same level of nodejs. And we do! Expect for the multi-query DB test which drops down to 70 req/sec :(

I am probably doing something stupid? There are two categories for DB testing: "raw" if we use plain JDBC and "normal" if we use a framework. Maybe I should just use plain JDBC?

  simon

Quick howto:

  $ mysql < config/create.sql
  $ ringo ringojs/ringo-main.js
  $ ab -n 1000 -c 10 http://localhost:8080/json
  $ ab -n 1000 -c 10 http://localhost:8080/db
  $ ab -n 1000 -c 10 http://localhost:8080/db?queries=20
 

[Michael Schwartz]

FWIW

When I implemented the MySQL driver for SilkJS, I found that the C++ calls to do the query took < 1ms and the nested loops to create an array of objects (one per row) took many milliseconds.  That's with all the optimizations I could come up with - like allocating the strings outside the loop just once, etc.

However, generating a string that was pure JSON and returning that to JS land and having the JS code do JSON decode, it was 35% faster.

Obviously Rhino is a different animal.  However, I looked at the Database code in Helma a lot and it suffers from considerable inefficiencies, and I think JDBC is probably not very efficient itself.

Cheers

--
You received this message because you are subscribed to the Google Groups "RingoJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ringojs+u...@googlegroups.com.
To post to this group, send email to rin...@googlegroups.com.
Visit this group at http://groups.google.com/group/ringojs?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Simon Oberhammer]

hm, interesting but on the other hand "servlet-raw" uses jdbc and is among top 3 in all DB benchs.

At least I'm sure I just chose the wrong tool for this benchmark job: ringo-sqlstore lives for its cache and that cache's not helping if you pick 20 from 10.000 and all the rows are tiny.

i wanted to see how a more raw jdbc does and switched the bench code to a forked version of ngv/sql-ringojs-client. I added connectionpooling by throwing some apachecommons jars at it. and *then* i checked the servlet-raw code and they just wrote a DbPoolServlet. So we could mimic them with a custom middleware and that would get rid of a lot of FactoryFactory ;)

anyhow, that's how the ringo bench app now (014c7fdfe5) looks:

https://github.com/oberhamsi/FrameworkBenchmarks/blob/master/ringojs/ringo-main.js

and it gives me pretty decent req/sec even for the multi-query bench.

 simon

[Brian Hauer]

Hi Simon!

We'd very much like to have RingoJS represented in the next round.  Let me know if we can help in any way or answer any questions you might have.  Incidentally, we just posted Round 3 at a new stand-alone site dedicated to the benchmarks:

http://www.techempower.com/benchmarks/#section=data-r3

We're going to be processing pull requests for a bit as we lead up to Round 4.

The most important factor for the multi-query test, as you've pointed out, is database connection pooling.  The worst performers will (re)connect to the database for every query.  And performance will remain less than ideal if the connection pool is too small.  Since these tests are intended to mimic a production deployment, do not hesitate to create a very large connection pool, as you would do for a production server.

For most frameworks, I recommend a pool of 256 or more database connections.  The reason being that our tests run at 256 client-side concurrency.  If you do any sort of server-side concurrency fan-out (e.g., you run your database queries asynchronously) you may benefit from even more database connections.  Basically, you want to do anything possible to prevent a request from having to wait for a connection to be available.

Also, I recommend using Wrk or WeigHTTP as a load tool rather than ApacheBench (ab).  ApacheBench is single-threaded and it can give unfairly low results for high performance frameworks and platforms.

To clarify: we use the "raw" suffix for any test that does not use an ORM or something ORM-like.  The Servlet test, for example, uses plain old JDBC, so it's considered raw.

[si...@nekapuzer.at]

Hi Brian,
thanks for the input.

> we use the "raw" suffix for any test that does not use an ORM

Okay, I renamed the db tests to ringojs-raw. I'll probably request a
pull within the next few days. I think I managed to cleanly integrate
into FrameworkBenchmarks.

Diff: <https://github.com/oberhamsi/FrameworkBenchmarks/compare/master>

Anything you don't like? I have to install two debian packages: ringo
itself and jsvc (we don't use jsvc for the bench but ringo.deb depends
on it and if I force to ignore it, apt-get becomes unhappy).

@everyone: I setup FrameworkBenchmarks on two EC2 t1.micro instances and
the tests are running now... I'm only running a subset of the benchmarks
so we should have results by tomorrow.

Maybe I'll try this on m1.large instances too, which is what TechEmpower
uses to gather results. Then we would know for sure how we stand for
only ~$30 :)

simon

If anyone wants to reproduce this: I recommend you use my "bare-minimum"
branch. You can install it with the normale README's instructions but
this branch only installs the software needed to run the java & nodejs
benchmarks.

<https://github.com/oberhamsi/FrameworkBenchmarks/tree/bare-minimum>

[Pat Falls]

Hi Simon,

I work with Brian on the benchmarks. Just so you know, the mico instances on EC2 are very hampered and probably will produce some unreliable results (I believe you'll run into CPU throttling), If I get some time today, I 'll try to run your branch on our dedicated hardware and let you know how it looks.

[Pat Falls]

Hi Simon,

I ran the tests on our i7 machines and got the following numbers:

JSON:     45k
DB:          31k
20-query: 4.9k 

[si...@nekapuzer.at]

Hi Pat, thanks!

indeed the micro is unusable but at least i got the maching image if i ever want
to run it on a bigger thing.

> JSON: 45k
> DB: 31k
> 20-query: 4.9k
>

that's alright. comparing with some selected others from round3
http://www.techempower.com/benchmarks/#section=data-r3


## servlet-raw
JSON: 218K
DB: 81K
20-query: 6K

## nodejs-raw
JSON: 72K
DB: 13K
20-query: 1K

## vertx
JSON: 125K
DB: 24K
20-query: 1K

## netty
JSON: 220K

all dedicated hardware, mysql.

simon

[Fabien Meghazi]

On Fri, Apr 19, 2013 at 4:24 PM, <si...@nekapuzer.at> wrote:
> Okay, I renamed the db tests to ringojs-raw. I'll probably request a
> pull within the next few days.

I told on irc that I would work on a branch but I failed to find time for that.
As we can do mysql and/or mongodb, I wanted to make a mongodb branch
with the java official driver that has connection pooling built-in.
But as I'm in the process of buying a house, selling an appartment,
that's very hard for me to find time, sorry about that.
So I'll do it for Round 5 if there's one.

--
Fabien Meghazi

Website: http://www.amigrave.com
Email: a...@amigrave.com
IM: amig...@gmail.com

[Brian Hauer]

Hi Fabien.  A RingoJS + Mongo test would be great.  I'm fairly certain there will be a Round 5.  :)

[si...@nekapuzer.at]

i added another test for ringojs-stick (our JSGI framework) and opened the pullrequest
https://github.com/TechEmpower/FrameworkBenchmarks/pull/176

fabien, i didn't see your irc message. i hang around #ringojs as oberhamsi. we can add mongodb anytime to what we have now, but i
want to get us in the next round :)

simon

[Fabien Meghazi]

On Sat, Apr 20, 2013 at 9:45 PM, <si...@nekapuzer.at> wrote:
> fabien, i didn't see your irc message. i hang around #ringojs as oberhamsi. we can add mongodb anytime to what we have now, but i
> want to get us in the next round :)

Of course ! Don't wait for me for the next round !
Ringo.js definitively needs press cover, benchmark cover, buzz cover,
... cover :-)

[Fabien Meghazi]

On Sun, Apr 21, 2013 at 12:36 PM, Fabien Meghazi <a...@amigrave.com> wrote:
> On Sat, Apr 20, 2013 at 9:45 PM, <si...@nekapuzer.at> wrote:
>> fabien, i didn't see your irc message. i hang around #ringojs as oberhamsi. we can add mongodb anytime to what we have now, but i
>> want to get us in the next round :)

I've got a forked repo here with a mongodb bench :

https://github.com/amigrave/FrameworkBenchmarks

It's completely untested for the moment, tomorrow I'll setup a vm in
order to run the tests.

[Fabien Meghazi]

On Mon, Apr 29, 2013 at 1:10 AM, Fabien Meghazi <a...@amigrave.com> wrote:
> I've got a forked repo here with a mongodb bench :
>
> https://github.com/amigrave/FrameworkBenchmarks

Ok, the mongodb bench should be working now :
https://github.com/amigrave/FrameworkBenchmarks

Gonna make a pull request

[Brian Hauer]

Hi Simon and Fabien,

Just a heads up that we're posting our Round 4 results tomorrow (May 2).  See my message about that at the following thread: https://groups.google.com/d/msg/framework-benchmarks/lW1hyXw4FG4/cRlQYiogcNQJ

(Sorry, posted the wrong link in the previous message.)

[Brian Hauer]

Hi Simon,

Just a heads-up: in Round 6, we're just adding a simple "plaintext" test at the request of several platform maintainers.  They've been asking for this one for a while.

Obviously we're welcoming anyone who has contributed tests in the past to add this new test.

More info: https://groups.google.com/d/msg/framework-benchmarks/XAzIlYrQvcg/qdqCKsoVIoYJ

[Simon Oberhammer]

> Just a heads-up: in Round 6, we're just adding a simple "plaintext" test at
> the request of several platform maintainers.  They've been asking for this
> one for a while.

thanks for the info. we'll surely have time to add this tiny test :)

i see the plaintext test will run at 16,384 concurrency... that will
be interesting

simon

[Brian Hauer]

Hi again Simon,

A quick update.  Pat is still working on several glitches in the Round 6 tests.  In the meantime I wanted to give you a heads up that we need all current database tests to be run without caches of any sort.  To that end, we're going to disable the query cache seen here:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/ringojs-convenient/app/models.js#L16

However, I wanted to quickly mention this to you in case this isn't in fact what we would consider a cache.  The key question is: does this cache allow SELECT operations to return a row without actually making a round-trip to the database server?  I suspect that is precisely what it does, and that is why we plan to disable it for the test.

A later test type will allow for the use of caches.

[Robert Gaggl]

hi brian,

(disclaimer: maintainer of ringo-sqlstore here): the query cache does not cache the result rows/values of database select statements, that would be the job of the entity cache of sqlstore - which isn't enabled in the ringojs-convenient benchmark app.

ringo-sqlstore uses a custom query language (eg. "select World.* from World where World.id = :id", as used in https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/ringojs-convenient/app/views.js#L25). these statements are parsed into an sql query string suitable for the backend database. what the query cache does is to cache these sql query strings - together with a collector used for creating model instances based on statement resultset values - but this still requires a round-trip to the database for retrieving rows (see https://github.com/grob/ringo-sqlstore/wiki/Caches) and creating the model instances.

if you want you can enable extensive logging by adding the following line to /usr/share/ringojs/modules/config/log4j.properties

log4j.category.ringo-sqlstore.lib.sqlstore.store = DEBUG

this will tell you exactly what sqlstore does (but please don't leave it enabled during benchmark runs ;)

robert

[si...@nekapuzer.at]

thanks for the info, but this not a a cache in your sense.

the "queryCache" only caches the parsing of SQL into a function tree. each
SELECT will still do a round-trip to the database.

>The key question is: does this cache allow
>SELECT operations to return a row without actually making a round-trip to
>the database server?

no, it does not :)

simon