MongoDB benchmarks

[diptamay]

Hello

I was just going through the following blog posts:
http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-yet-another-silly.html
http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-fetch-by-secondary.html
http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-update-performance.html

And, as per the numbers, MongoDB doesn't give great numbers compared
to MySQL (used to store/retrieve K-V data). Additionally, the author
claims that there are mutex contentions when running the tests on a 16-
core server.

I was just wondering, what the MongoDB team feels about it, like are
the numbers valid? Were the tests done in the best way as it could
have been done? Ideas/thoughts on the mutex contention issue raised?

Cheers!
Diptamay

[Eliot Horowitz]

The benchmarks are valid - though are for a very specific case (hence
why he says silly).
On very simple cases like that - speed is more of factor of lots of
tiny code optimizations which we haven't done yet.
Mysql is mature and is highly optimized so it shouldn't be too
shocking that you can make it perform very well for some cases.

We've been focused on more real world performance considerations that
our users are seeing.

For example - because you can embed objects in mongo, you can avoid
doing a join in mysql.
So you may want to benchmark a single document load in mongo where you
have to load 10 rows in mysql.

We definitely will be doing more optimization to get those kind of
numbers higher, but you need to look at performance as a whole.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[MARK CALLAGHAN]

Hello world.

The micro-benchmarks are extremely valid to me. But they are
micro-benchmarks not system benchmarks and I doubt they are critical
for most other deployments today. Not everyone is doing
high-throughput OLTP on 8 and 16 core servers right now. MongoDB did
better than I expected. It will probably be easy to fix the mutex
contention for the read-only tests and if I can only figure out how to
build mongodb on my not-so-current platform (other post pending for
that) then I will try that out.

These are the first tests I run when looking at new MySQL upgrades.
They are good at finding obvious problems. When I know there are few
obvious problems then I can move on to more interesting workloads. I
prefer not to wonder whether there are obvious performance bottlenecks
when I am debugging complex performance problems in production.

--
Mark Callaghan
mdca...@gmail.com

[Eliot Horowitz]

Yes - there are places where under massive concurrency there can be
mutex contention issues.
Should be pretty easy to fix for those cases though.

On Tue, Sep 14, 2010 at 11:05 AM, diptamay <dipt...@gmail.com> wrote:
> The numbers are about reading/storing K-V data. So, yes benchmarks
> were geared toward that case specifically. What I am more interested
> in, is the mutex contention issue. Any ideas on that?
>
> -Diptamay

>
> On Sep 14, 10:48 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:
>> The benchmarks are valid - though are for a very specific case (hence
>> why he says silly).
>> On very simple cases like that - speed is more of factor of lots of
>> tiny code optimizations which we haven't done yet.
>> Mysql is mature and is highly optimized so it shouldn't be too
>> shocking that you can make it perform very well for some cases.
>>
>> We've been focused on more real world performance considerations that
>> our users are seeing.
>>
>> For example - because you can embed objects in mongo, you can avoid
>> doing a join in mysql.
>> So you may want to benchmark a single document load in mongo where you
>> have to load 10 rows in mysql.
>>
>> We definitely will be doing more optimization to get those kind of
>> numbers higher, but you need to look at performance as a whole.
>>
>>
>>
>> On Tue, Sep 14, 2010 at 10:40 AM, diptamay <dipta...@gmail.com> wrote:
>> > Hello
>>
>> > I was just going through the following blog posts:

>> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-yet-another-...
>> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-fetch-by-sec...
>> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-update-perfo...

[diptamay]

The numbers are about reading/storing K-V data. So, yes benchmarks
were geared toward that case specifically. What I am more interested
in, is the mutex contention issue. Any ideas on that?

-Diptamay

On Sep 14, 10:48 am, Eliot Horowitz <eliothorow...@gmail.com> wrote:

> The benchmarks are valid - though are for a very specific case (hence
> why he says silly).
> On very simple cases like that - speed is more of factor of lots of
> tiny code optimizations which we haven't done yet.
> Mysql is mature and is highly optimized so it shouldn't be too
> shocking that you can make it perform very well for some cases.
>
> We've been focused on more real world performance considerations that
> our users are seeing.
>
> For example - because you can embed objects in mongo, you can avoid
> doing a join in mysql.
> So you may want to benchmark a single document load in mongo where you
> have to load 10 rows in mysql.
>
> We definitely will be doing more optimization to get those kind of
> numbers higher, but you need to look at performance as a whole.
>
>
>
> On Tue, Sep 14, 2010 at 10:40 AM, diptamay <dipta...@gmail.com> wrote:
> > Hello
>
> > I was just going through the following blog posts:

> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-yet-another-...
> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-fetch-by-sec...
> >http://mysqlha.blogspot.com/2010/09/mysql-versus-mongodb-update-perfo...
>

[Eliot Horowitz]

Not sure when - may look soon.
But are you sure its relevant? According to Mark's graphs, you need
about 64 concurrent reads before it gets in the way.
And even then it flattens out, doesn't really drop off very much
(maybe at 120 concurrent reads)
At that level - he's doing around 60k reads/second.
Are you going to be doing that kind of read levels?
Object size is also a pretty significant factor for total throughput.

On Tue, Sep 14, 2010 at 11:13 AM, diptamay <dipt...@gmail.com> wrote:
> We are looking at getting 8 core servers (with hyper-threading enabled
> - taking them to 16 virtual cores) for MongoDB, so the benchmarks/
> mutex contention are of specific importance. Any ideas by when you
> could have those optimizations in place.
>
> -Diptamay

[diptamay]

We are looking at getting 8 core servers (with hyper-threading enabled
- taking them to 16 virtual cores) for MongoDB, so the benchmarks/
mutex contention are of specific importance. Any ideas by when you
could have those optimizations in place.

-Diptamay

[diptamay]

Not of immediate concern, but it helps us in planning better. We might
need that kind of read levels sometime down the line but not
immediately, so was trying to find out the plan. Our object size on an
average is around 20-30 KB.

Thanks
Diptamay

[Eliot Horowitz]

So - just to sanity check.
60k fetches/second on a 20kb object is over 1 gigabyte/second.
So if you've got a normal gigabit network connection - the db can do
almost 10 times more than the network can handle.
Even with a 10gbps network card - you're still probably going to hit
network bounds...

[MARK CALLAGHAN]

Even better, when I finally realized that the client host was
saturated by pymongo during the get by primary key test I used two
16-core client hosts and doubled peak QPS for MongoDB --
http://mysqlha.blogspot.com/2010/09/oh-no-mongodb-can-be-fast-for-key-value.html

That is an impressive result.

The results for get by secondary key and update by secondary key are
still valid though.

--
Mark Callaghan
mdca...@gmail.com

[Fabio Kaminski]

why all these benchmarks we found published are in python or in ruby.. the language and the implementation itself could be a major bottleneck..

it may be good to test how python or ruby behaves with the database used.. but, the driver interface beetween db and application must be a major concern, and i think for good results, you must use something provided by someone who has a good understanding of the db.. with preference in the same platform..  wich for both would be c/c++

if you read for instance the c++ provided driver  for mongo and the "rest" ..it make it obvious the others "hackish" implementations (and thats true for mysql too)..

on thing that leverage my curiosity is how a stateless version of mongo would behave.. like a rest, or something with a skinny protocol...

i think it would loose in the short term.. with less clients.. but it would scale better.. and web servers are there to prove that..

and how about vs. google v4 patch? :)