[morphia] performance
Brian Nesbitt
I have been following morphia, playframework (http://
www.playframework.org/) and mongo for awhile and currently use them
all.
I helped with the Mongo module for the PlayFramework, but saw too much
duplication with Morphia so I am now finishing up a "mongoViaMorphia"
module for play.
Not totally unrelevant, I have been reading Scott's and Ian's
conversations about performance and possibly using lambda expressions.
Read an interesting article yesterday on Morphia.
http://www.scalabiliti.com/blog/mongodb_and_morphia_performance
Seems as though it was posted about a month ago, but was using an
older version of Morphia.
Anyway, I decided to reattempt their "performance" tests to see if its
better now. I also added a 3rd test that used the DAO interface.
My config:
core i7 860, 8 GIGs ram and mongod (1.4.4 with logging on) is running
from SSD.
I ran each test 6 times (dropping the first test result as I would
drop the collection before each run and the first run was always
higher since I think it was allocating the block of space from disk...
each run after that was very consistent showing less than 5 ms of
variation
Test Records Speed
----------------------------------------------------------------------------
BasicDBObject (Insert) 10,000 152 ms
Morphia (only toDBObject) 10,000 158 ms
Morphia Datastore 10,000 1082 ms
Morphia DAO 10,000 1150 ms
I threw in the datastore and DAO test not expecting a difference as
its fairly light from what I have seen. I expected the overhead to be
in the morphia.toDBObject() function which all morphia tests would be
calling.
After about 10 more seconds I thought, maybe the Datastore and DAO was
forcing the getLastError() call so it was actually doing round trips
to local host rather than a fire n' forget.
After a bit of digging around I don't think Write concern is exposed
easily (scott mentioned next on the list??), so I went the other way
and turned it on for all. Here are the new numbers.
Test (with getLastError()) Records Speed
---------------------------------------------------------------------------
BasicDBObject (Insert) 10,000 674 ms
Morphia (only toDBObject) 10,000 673 ms
Morphia Datastore 10,000 1082 ms
Morphia DAO 10,000 1150 ms
Any ideas why the Datastore/DAO is still much longer??? It's code
doesn't seem like it would have added that much overhead.
Could that be just the pre and post save operations (fyi I had non
registered) ??
I still think this overhead more than justifies what Morhpia brings to
the table!
I for one am moving full steam ahead with PlayFramework,
mongoViaMorphia, morphia and Mongo!!
Scott Hernandez
Reads are much, much closer to the performance of the raw driver. In a
lot of my tests the difference was measures in milliseconds, and
sometimes not even into the tens. Writes really have more issues, as
you will see below.
On Sun, Jul 18, 2010 at 12:20 PM, Brian Nesbitt <sport...@gmail.com> wrote:
> Background:
> I have been following morphia, playframework (http://
> www.playframework.org/) and mongo for awhile and currently use them
> all.
> I helped with the Mongo module for the PlayFramework, but saw too much
> duplication with Morphia so I am now finishing up a "mongoViaMorphia"
> module for play.
Sounds good.
> Not totally unrelevant, I have been reading Scott's and Ian's
> conversations about performance and possibly using lambda expressions.
>
> Read an interesting article yesterday on Morphia.
> http://www.scalabiliti.com/blog/mongodb_and_morphia_performance
>
> Seems as though it was posted about a month ago, but was using an
> older version of Morphia.
Yep, that is very old code :) (any time you see @MongoDocument, you
should think ancient -- every time you used that code it basically
reflected on the class per use.)
> Anyway, I decided to reattempt their "performance" tests to see if its
> better now. I also added a 3rd test that used the DAO interface.
I assume you used 0.94?
> My config:
> core i7 860, 8 GIGs ram and mongod (1.4.4 with logging on) is running
> from SSD.
> I ran each test 6 times (dropping the first test result as I would
> drop the collection before each run and the first run was always
> higher since I think it was allocating the block of space from disk...
> each run after that was very consistent showing less than 5 ms of
> variation
>
> Test Records Speed
> ----------------------------------------------------------------------------
> BasicDBObject (Insert) 10,000 152 ms
> Morphia (only toDBObject) 10,000 158 ms
> Morphia Datastore 10,000 1082 ms
> Morphia DAO 10,000 1150 ms
>
> I threw in the datastore and DAO test not expecting a difference as
> its fairly light from what I have seen. I expected the overhead to be
> in the morphia.toDBObject() function which all morphia tests would be
> calling.
> After about 10 more seconds I thought, maybe the Datastore and DAO was
> forcing the getLastError() call so it was actually doing round trips
> to local host rather than a fire n' forget.
Yep, the WriteConcern is one of the issues when doing these
comparisons. The code correctly forces Strict so that we can throw
exceptions on duplicate inserts/save/updates, and if there are
problems during writes. In addition there are extra getLastError()
calls since the lower level driver calls it, and then we call it. I
have patches submitted to help remove these repeated call by using a
cached results from the lower level driver if it has already been
called.
I think the default of strict makes sense since most people expect to
get exceptions, but we will have to enable a way to turn that off
(waiting for more control in the next release of the driver).
At that point, with the 2.1 driver, we should be able to remove all
the extra overhead for calls (extra getLastError ones at least) and be
able to align WriteConcerns (with the issue that server errors won't
get raised if Strict isn't used -- which you will have to opt into). I
hope that at that point we can come close to within 20-30 overhead
compared to the raw driver.
For a long time in the 0.93-snapshots we were down to less than 10%
overhead (not forcing Strict or making extra calls) but then we
re-factored a lot of code and introduced the lazy (with proxy) code.
I'm pretty sure most of the performance degradation we saw was related
to the extra calls and strict use for exceptions (and reporting --
like in UpdateResults), but some came from the new code as well. I
have optimized some of the code and was about to drop an order of
magnitude in some tests (that is now in 0.94). Once we move to new
driver we should be able to profile our code and see just what kind of
overhead Morphia adds.
> After a bit of digging around I don't think Write concern is exposed
> easily (scott mentioned next on the list??), so I went the other way
> and turned it on for all. Here are the new numbers.
>
> Test (with getLastError()) Records Speed
> ---------------------------------------------------------------------------
> BasicDBObject (Insert) 10,000 674 ms
> Morphia (only toDBObject) 10,000 673 ms
> Morphia Datastore 10,000 1082 ms
> Morphia DAO 10,000 1150 ms
>
> Any ideas why the Datastore/DAO is still much longer??? It's code
> doesn't seem like it would have added that much overhead.
There shouldn't be so much difference in the Datastore versus DAO
code; that should be easy to figure out and optimize.
> Could that be just the pre and post save operations (fyi I had non
> registered) ??
It is possible, but this should only present a slowdown if there are
event handlers for many classes.
> I still think this overhead more than justifies what Morhpia brings to
> the table!
That is good to hear, and as I've said I hope to be able to get it
down to less than a 25% overhead on writes.
> I for one am moving full steam ahead with PlayFramework,
> mongoViaMorphia, morphia and Mongo!!
Good to hear, let us know if you need any help along that way; I'm
sure you will :)
Brian Nesbitt
> Please keep in mind all these numbers are for writes, not reads.
>
performance.... my motivation point more was to rerun the test from
that article and show its better now and really not that bad.
> I assume you used 0.94?
>
> Yep, the WriteConcern is one of the issues when doing these
> comparisons. The code correctly forces Strict so that we can throw
> exceptions on duplicate inserts/save/updates, and if there are
> problems during writes. In addition there are extra getLastError()
> calls since the lower level driver calls it, and then we call it. I
> have patches submitted to help remove these repeated call by using a
> cached results from the lower level driver if it has already been
> called.
>
> I think the default of strict makes sense since most people expect to
> get exceptions, but we will have to enable a way to turn that off
> (waiting for more control in the next release of the driver).
for default not durable on a single server.
> At that point, with the 2.1 driver, we should be able to remove all
> the extra overhead for calls (extra getLastError ones at least) and be
> able to align WriteConcerns (with the issue that server errors won't
> get raised if Strict isn't used -- which you will have to opt into). I
> hope that at that point we can come close to within 20-30 overhead
> compared to the raw driver.
>
getting double calls, that would explain the time difference? Seems
like its about the right timing difference as well??
>
> That is good to hear, and as I've said I hope to be able to get it
> down to less than a 25% overhead on writes.
>
> > I for one am moving full steam ahead with PlayFramework,
> > mongoViaMorphia, morphia and Mongo!!
>
> Good to hear, let us know if you need any help along that way; I'm
> sure you will :)
Scott Hernandez
>
>> Please keep in mind all these numbers are for writes, not reads.
>>
>
> Reads are for wimps... lol... yes, agreed, it is a very biased
> performance.... my motivation point more was to rerun the test from
> that article and show its better now and really not that bad.
Yes, yes... I just wanted to remind people that this only one part of
your data access, the writes. In fact, just really saves and inserts.
>> I assume you used 0.94?
> Yes, I was using 0.94.
>
>> Yep, the WriteConcern is one of the issues when doing these
>> comparisons. The code correctly forces Strict so that we can throw
>> exceptions on duplicate inserts/save/updates, and if there are
>> problems during writes. In addition there are extra getLastError()
>> calls since the lower level driver calls it, and then we call it. I
>> have patches submitted to help remove these repeated call by using a
>> cached results from the lower level driver if it has already been
>> called.
>>
>> I think the default of strict makes sense since most people expect to
>> get exceptions, but we will have to enable a way to turn that off
>> (waiting for more control in the next release of the driver).
>
> Agreed... default STRICT makes sense. A lot of people put mongo down
> for default not durable on a single server.
Even with strict you still don't get durability. In 1.8 we should get
something better with an append log; you will write immediately to
that log, and then flush and compare somewhere down the road, or
something like that.
However there will be real WriteConcern class where you can specify
you want the write to get flushed to disk before the return; that will
help a bit for paranoid people.
>> At that point, with the 2.1 driver, we should be able to remove all
>> the extra overhead for calls (extra getLastError ones at least) and be
>> able to align WriteConcerns (with the issue that server errors won't
>> get raised if Strict isn't used -- which you will have to opt into). I
>> hope that at that point we can come close to within 20-30 overhead
>> compared to the raw driver.
>>
>
> If the underlying driver is calling getLastError() as well and its
> getting double calls, that would explain the time difference? Seems
> like its about the right timing difference as well??
Yes, it explains a bit of the difference; reducing the calls with get
us closer.
I don't want performance to be a reason to chose the raw mongodb driver.
Brian Nesbitt
STRICT and secondly to remove the secondary call to getLastError()
when STRICT is on to get more valid numbers (I did confirm the double
call as you suggested). The initial numbers are looking much better
more in line with what is expected. I'll post them up before end of
week.
Brian Nesbitt
WriteConcern = Strict
* Only underlying driver calling getLastError(), Morphia check in
save() commented out
* Testing with 15 executions of 10000 documents
Basic Driver, total time 10055ms, avg/execution 670ms, avg/document
0ms
Morphia toDBObject, total time 9515ms, avg/execution 634ms, avg/
document 0ms
Morphia Datastore, total time 10425ms, avg/execution 695ms, avg/
document 0ms
Morphia DAO, total time 10400ms, avg/execution 693ms, avg/document 0ms
WriteConcern = None
* Morphia call to getLastError() in in save() commented out
* Morphia setting WriteConcern.STRICT in
DatastoreImpl.getCollection() commented out
* Testing with 15 executions of 10000 documents
Basic Driver, total time 2530ms, avg/execution 168ms, avg/document 0ms
Morphia toDBObject, total time 2465ms, avg/execution 164ms, avg/
document 0ms
Morphia Datastore, total time 2659ms, avg/execution 177ms, avg/
document 0ms
Morphia DAO, total time 2448ms, avg/execution 163ms, avg/document 0ms
WriteConcern = None test was done for completeness but doesn't mean
much.
The real test was the first one comparing the Datastore/DAO to Basic
Driver which it looks like morphia is only providing approx a 4%
overhead!!
Needless to say these are pretty impressive times from morphia and
mongo.