Preview of round 8

[Finn Bock]

Hi,

preliminary results from the techempower benchmark show that vertx goes from 7,179 to 34,678 request/sec on the "Single query" test. This is using 2.1M1 and is without the lastest speedup to local eventbus addresses.

In json and plaintext tests vertx is still one of the very fastest.

http://www.techempower.com/benchmarks/previews/round8/

regards,
Finn

[Tim Fox]

On 26/11/13 07:43, Finn Bock wrote:

Hi,

preliminary results from the techempower benchmark show that vertx goes from 7,179 to 34,678 request/sec on the "Single query" test. This is using 2.1M1 and is without the lastest speedup to local eventbus addresses.

Finn, do you have your changes for the a) Json b) native Mongo driver? I would like to work on this, this week.

In json and plaintext tests vertx is still one of the very fastest.

http://www.techempower.com/benchmarks/previews/round8/

regards,
Finn

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Finn Bock]

[Tim Fox]:

Finn, do you have your changes for the a) Json b) native Mongo driver? I would like to work on this, this week.

I can't say I have anything further for json deep copy, but the work-in-progress for the mongo driver can be found here:

https://github.com/bckfnn/vertx-native-mongo

regards,
Finn

[Tim Fox]

I've significantly increased the perf of the db test by not using the event bus at all. The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

I'm still interested in your native driver when it matures a bit, as we need a proper async driver anyway, and if we can get further performance boosts, that's a bonus :)

[Finn Bock]

[Tim Fox]:

I've significantly increased the perf of the db test by not using the event bus at all. The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Unfortunately I can't replicate your finding. I tested your version at https://github.com/purplefox/FrameworkBenchmarks.
The mongo tests ran 2-3 times slower and the json test 10 times slower.

I had to add a -cp argument to include the mongo lib, I think that is missing from your version.

regards,
Finn

[Tim Fox]

On 27/11/13 14:16, Finn Bock wrote:

[Tim Fox]:

I've significantly increased the perf of the db test by not using the event bus at all. The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Unfortunately I can't replicate your finding. I tested your version at https://github.com/purplefox/FrameworkBenchmarks.
The mongo tests ran 2-3 times slower and the json test 10 times slower.

That's odd - I haven't really made any changes to the JSON test

I had to add a -cp argument to include the mongo lib, I think that is missing from your version.

regards,
Finn

[Tim Fox]

On 27/11/13 14:16, Finn Bock wrote:

[Tim Fox]:

I've significantly increased the perf of the db test by not using the event bus at all. The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Unfortunately I can't replicate your finding. I tested your version at https://github.com/purplefox/FrameworkBenchmarks.
The mongo tests ran 2-3 times slower and the json test 10 times slower.

I'm a bit confused as to why you should see slower results. Just to clarify - then you say 'slower' do you mean slower than before my pull request? Or slower compared to something else.

Also... what OS are you using? Can you give details of the OS and machine setup?

I had to add a -cp argument to include the mongo lib, I think that is missing from your version.

regards,
Finn

[Finn Bock]

On 27/11/13 14:16, Finn Bock wrote:

[Tim Fox]:

I've significantly increased the perf of the db test by not using the event bus at all. The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Unfortunately I can't replicate your finding. I tested your version at https://github.com/purplefox/FrameworkBenchmarks.
The mongo tests ran 2-3 times slower and the json test 10 times slower.

I'm a bit confused as to why you should see slower results. Just to clarify - then you say 'slower' do you mean slower than before my pull request? Or slower compared to something else.

Slower than before the pull request.
 

Also... what OS are you using? Can you give details of the OS and machine setup?

Ahh, yes, well. Its a windows 7 machine i7 where I run mongo, vertx and WeigHTTP, so the numbers I get should not be compared with anything.

Still I got 100.000 req/sec before your pull request and 40.000 req/seq after your PR on the json test. If I run a mongo test first that open 660 connections to mongo, and then run the json, it drops to 9.000 req/seq.

Perhaps the explanation is that while WebServer.start() is running on a vert.x-worker-thread, the WebServer.handle(req) method is still called from a vert.x-eventloop-thread.

Whatever the reason may be, I just wanted to make sure that on linux you don't see this slowdown.

regards,
Finn

[Brice Dutheil]

The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Wouldn't that be defeating the purpose of showing performance of vert.x using the eventbus and standard verticles ? So people could expect comparable results with their own vert.x implementation, and maybe learn how to do things properly.

Cheers,

-- Brice

[Finn Bock]

[Brice Dutheil]:

The benchmark now uses the Mongo DB driver API directly and the main verticle is marked as a worker verticle.

Wouldn't that be defeating the purpose of showing performance of vert.x using the eventbus and standard verticles ? So people could expect comparable results with their own vert.x implementation, and maybe learn how to do things properly.

The benchmark have a metadata field :

approach (metadata): Realistic or Stripped

And maybe is should be set to Stripped when too many tricks and shortcuts are used.

Even a stripped test is usefull as an indicator of the upper limit a framework can achieve. So IMO is does not defeat the purpose.

regards,
Finn

[Tim Fox]

Sigh.. the problem was I was running the benchmark on a crappy VM. When I run it on a real machine I get similar results to you for plaintext and json.

Most of the overhead in the db test seems to be due to the various JsonObjects that are created and encoded/decoded.

I also tried your native driver and I can get much better performance when using that :) So this is the way to go, but it really needs a bit more work before it can be used from the benchmark. I'm tempted to just fork it and spend tomorrow hacking away at it. Hmmm....

[Paulo Lopes]

just throwing out some ideas, why do we need the jsonobject class and not just use java.util.Map and java.util.List?

I mean when i use groovy the api takes maps ans lists, when i code in js i can also put any object such as dates into json

and it seems simple enough and does not need so many checks and copies all around...

Why do we need a specialized class for java? I know that one can use maps and list and only emcapsulate as jsonobject when needed but i think if we would use the other way around first maps and lists and jsoobject for controlled type safety it could (eventually) increase performance of the whole system...

[petermd]

Alternatively you could look to seperate the Builder pattern from the runtime Object. I think it's difficult to optimize a general purpose random-access mutable object even though I imagine a lot of use-cases are much more linear. Whereas a streaming builder can be implemented to render direct to the target format (binary Buffer for Json or BSON for mongo etc) and should be much more memory efficient.

I use a standard JsonBuilder interface with a concrete implementation for JsonObject but plan to provide additional implementations that render to BSON, Buffer etc. If the client Verticle can render the DB objects as raw BSON then they could (in theory!) be transported across the EventBus and directly onto the Mongo Client socket without any codec steps (although there may be some wrapping/unwrapping of request envelopes)

-Peter

[Tim Fox]

On 28/11/13 16:45, petermd wrote:

Alternatively you could look to seperate the Builder pattern from the runtime Object. I think it's difficult to optimize a general purpose random-access mutable object even though I imagine a lot of use-cases are much more linear. Whereas a streaming builder can be implemented to render direct to the target format (binary Buffer for Json or BSON for mongo etc) and should be much more memory efficient.

I use a standard JsonBuilder interface with a concrete implementation for JsonObject but plan to provide additional implementations that render to BSON, Buffer etc. If the client Verticle can render the DB objects as raw BSON then they could (in theory!) be transported across the EventBus and directly onto the Mongo Client socket without any codec steps (although there may be some wrapping/unwrapping of request envelopes)

Nice idea

-Peter

On Thursday, 28 November 2013 16:25:37 UTC, Paulo Lopes wrote:

just throwing out some ideas, why do we need the jsonobject class and not just use java.util.Map and java.util.List?

I mean when i use groovy the api takes maps ans lists, when i code in js i can also put any object such as dates into json

and it seems simple enough and does not need so many checks and copies all around...

Why do we need a specialized class for java? I know that one can use maps and list and only emcapsulate as jsonobject when needed but i think if we would use the other way around first maps and lists and jsoobject for controlled type safety it could (eventually) increase performance of the whole system...

[Paulo Lopes]

If you need help with BSON i can assist with code from mod-bson-io

Cheers,
Paulo

[Tim Fox]

I've also made some significant performance improvements by a more sensible conversion between JsonObject and BasicDBObject in mongo persistor, and by lazily copying JsonObject over the event bus.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

[Paulo Lopes]

If the idea of making a Builder interface is to go forward please, please :) follow what the JS JSON object does :)

encoding of base types:

* null
* number (if finite, in case of infinity then null - https://github.com/douglascrockford/JSON-js/blob/master/json2.js#L264)
* boolean
* string
* array []
* object {}
* date (encoded as ISO8601 UTC string - https://github.com/douglascrockford/JSON-js/blob/master/json2.js#L174)
* all others (not encoded at all, unless inside an array then it their encoding is 'null')

it would be even more awesome if there was a String: toJSON() interface than in that case use that as that string would be the value... https://github.com/douglascrockford/JSON-js/blob/master/json2.js#L240
but i can live without it :)

Cheers,
Paulo

[Finn Bock]

[Tim Fox]:

I've also made some significant performance improvements by a more sensible conversion between JsonObject and BasicDBObject in mongo persistor, and by lazily copying JsonObject over the event bus.

It's great. I get a 15% faster multi query test.

What is the reason for your change to the vertx.pool.eventloop.size default? And even with the change I only get 10 eventloop threads. And mine are numbered 0 to 10 but number 2 is always missing. Strange !

BTW, last day for submission of patches for round 8 is monday.

regards,
Finn

[Brice Dutheil]

Hi Finn,

Indeed the benchmark have different load distributions. But my point was focused on the vert.x verticle code used in the benchmark, I think benchmark reviewers will like to see actual modules and separation of concerns where appropriate.

On another topic, the builder pattern is indeed appealing.

Cheers,

-- Brice

[Tim Fox]

On 29/11/13 08:51, Finn Bock wrote:

[Tim Fox]:

I've also made some significant performance improvements by a more sensible conversion between JsonObject and BasicDBObject in mongo persistor, and by lazily copying JsonObject over the event bus.

It's great. I get a 15% faster multi query test.

What is the reason for your change to the vertx.pool.eventloop.size default?

It doesn't make a lot of difference, but the reasoning is, if you chose number of event loops = number of cores, then the OS doesn't *always* get it right and distribute them across all cores so you might have more than one on one core, and none on others. Java doesn't have a way of pinning threads to cores so if you increase the number of loops the probability of them being poorly distributed is less. That's the theory anyway! But it doesn't do a lot of harm to have some more.

And even with the change I only get 10 eventloop threads. And mine are numbered 0 to 10 but number 2 is always missing. Strange !

BTW, last day for submission of patches for round 8 is monday.

Yep, Norman and I were up late last night and are continuing to work :)

regards,
Finn

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[El Gazali]

I just hope that Vert.x is ready to go for round 8?

[Norman Maurer]

I think it is ready but expect more performance improvements. Already working on this ;)

 

-- 
Norman Maurer

On 5. Dezember 2013 at 20:08:54, El Gazali (elhosin...@gmail.com) wrote:

I just hope that Vert.x is ready to go for round 8?

[El Gazali]

Great stuff :)

[El Gazali]

Seeing this expensive NPE makes me feel uncomfortable. I really hope it is sorted.

https://github.com/TechEmpower/TFB-Round-8/blob/master/i7/preview-2/latest/logs/vertx/err.txt

On Thursday, 5 December 2013 19:12:39 UTC, Norman Maurer wrote: