I did some tests for insert performance of MongoDB. It shows insert performance is slower when collection have index and SAFE write concern is set.
Here is the test result
* Inserting with no index and safe write concern not set threads count average speed 1 10161 2 17519 3 15859 (thread count means the concurrent inserting thread count) The speed seem is higher . * Inserting with index and safe write concern not set threads count average speed 1 6320 2 6240
* Inserting with index and safe write concern set threads count average speed 1 972 2 1530 5 2573
The data listed seems normal? the speed of inserting with index and safe write concern, is only 1/5 of that with no index and no safe write concern. Is any way to speed up the performance for inserting with index and safe write concern condition? I only create two index for that collection.
Yes - there will be a huge impact to performance when moving from "NONE" to "SAFE" write concern.
The reason for the impact is simple. In the "NONE" case the driver simply gets the message on the wire and returns to the caller. In the "SAFE" case you need to wait for the driver to respond to the insert. The round trip latency is usually orders of magnitude longer than pushing the bytes into a socket even when connected to a MongDB server on the same machine.
My team has written a new Java driver to allow developers to overcome this exact problem by using asynchronous method calls. To get the full benefit the developer needs to do some work to either handle responses via a callback or defer processing results via a Future. http://www.allanbank.com/mongodb-async-driver/
We have run a custom micro-benchmark to show the relative performance on using the different write concerns. (For our driver Durability is the same as the 10gen's WriteConcern and we call "SAFE", "ACK" since we feel that is a more descriptive name.)
In addition, the way the 10gen driver only allows a single thread use a connection causes scalability problems as demonstrated using the YCSB. Our driver does not have the same scalability problems since multiple threads can be using a single connection concurrently. http://www.allanbank.com/mongodb-async-driver/performance/ycsb.html
If you have a specific question about using our driver feel free to ask on this list or via direct email.
So a few different things are going on when you perform an insert, and it is also very important to know the details of write concern.
1. Write Concern When you set a safe write concern, the application blocks and waits for a response from the server confirming that it received the insertion command. When you do not set a write concern, the application just sends the message and returns, not awaiting any response from the server (you can think of it as fire and forget from an application perspective). So, to be clear, it is not that inserting without a write concern is "faster" but rather that it does not wait for confirmation that data has been written and returns so it does not have to deal with server processing time and network latency of the messages.
2. Inserting with an index
Now, if you do not have an index and perform an insert, then all that happens is your insert goes to the server and gets written in the next allocated block of the disk. Things like fragmentation and growing file sizes can complicate this process as you may need to move files on disk in order to accommodate their size, so things like the state of your disk and the size of the files can invalidate many of these "performance tests". But that aside, there is only one write operation and then the server returns a response if there is a write concern.
Now then, If you perform an insert with an index than you are actually performing 2 write operations and a read. 1 write to insert your new document, 1 read to page the index into memory if it is not there already, and another write to add this new document to the index. And then if you are running with a write concern it will return a message to the application confirming its been completed. So yes, having an index means slower write operations in exchange for much faster reads. This will get even slower if you have multiple indexes, because you will have to update all of them on every insert.
3. What does this all mean:
What this effectively means is that if you run application side with no write concern, then your tests will always result consistant values whether there as an index or not because you send out the write message but never actually wait for it to complete and get confirmation. However if you ARE running with a write concern of 1, then you must wait for the operation to actually be performed and receive a response. With an index you are performing more operations on insert than without one, so it will naturally be slower for you to complete and receive a confirmation. Even more so if you run with higher write concerns for replication.
There really isn't any real way to speed up inserts from an application side if you want to wait for confirmation from the server that the insert was successful. And there's no real way to make inserting with an index any faster because you cant get around the fact that all of these updates on the index need to be performed with every insertion. Hopefully why this is, makes sense now.
On Saturday, September 29, 2012 4:08:39 AM UTC-4, Hanson Lu wrote:
> I did some tests for insert performance of MongoDB. It shows insert > performance is slower when collection have index and SAFE write concern is > set.
> Here is the test result
> * Inserting with no index and safe write concern not set > threads count average speed > 1 10161 > 2 17519 > 3 15859 > (thread count means the concurrent inserting thread count) > The speed seem is higher . > * Inserting with index and safe write concern not set > threads count average speed > 1 6320 > 2 6240
> * Inserting with index and safe write concern set > threads count average speed > 1 972 > 2 1530 > 5 2573
> The data listed seems normal? the speed of inserting with index and > safe write concern, is only 1/5 of that with no index and no safe write > concern. > Is any way to speed up the performance for inserting with index and > safe write concern condition? > I only create two index for that collection.
In the shell, I can use db.getCollectionNames() get a array of collection's name, but I don't know how to get collectionNames by c driver, can somebody help me ?
In short: there is no getCollectionName in the c drivers. Dunno if the
drivers have been updated with this function yet. But as you can read in
the post, it is quite easy to implement it at you own. :-)
*"There is no getCollectionNames command. If you type db.getCollectionNames
in the shell (note no parens) it will print out the source and you can see
how it's implemented there."
*
On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipingz...@gmail.com> wrote:
> Hi all:
> In the shell, I can use db.getCollectionNames() get a array of
> collection's name, but I don't know how to get collectionNames by c driver,
> can somebody help me ?
> Bruce Zhao
> 2012-10-08
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@**googlegroups.com<mongodb-user%2Bunsubscribe@goog legroups.com>
> See also the IRC channel -- freenode.net#mongodb
Hi Osmar, I want to note that the inserting speed is seen from client side and server side(mongostat). The both is almost same.
BTW, the result of insering with safe write conern , and no index. Thread average speed 5 4156 10 4898
So if the speed with index and safe concern of my test is normal, is MongoDB not suitable for apps that require high inserting speed? It seems that there is no big improvement than Relational Db when inserting with index and safe conern.
On Saturday, September 29, 2012 4:08:39 PM UTC+8, Hanson Lu wrote: > I did some tests for insert performance of MongoDB. It shows insert > performance is slower when collection have index and SAFE write concern is > set.
> Here is the test result
> * Inserting with no index and safe write concern not set > threads count average speed > 1 10161 > 2 17519 > 3 15859 > (thread count means the concurrent inserting thread count) > The speed seem is higher . > * Inserting with index and safe write concern not set > threads count average speed > 1 6320 > 2 6240
> * Inserting with index and safe write concern set > threads count average speed > 1 972 > 2 1530 > 5 2573
> The data listed seems normal? the speed of inserting with index and > safe write concern, is only 1/5 of that with no index and no safe write > concern. > Is any way to speed up the performance for inserting with index and > safe write concern condition? > I only create two index for that collection.
> In short: there is no getCollectionName in the c drivers. Dunno if the > drivers have been updated with this function yet. But as you can read > in the post, it is quite easy to implement it at you own. :-)
> /"There is no getCollectionNames command. If you type > db.getCollectionNames in the shell (note no parens) it will print out > the source and you can see how it's implemented there."
> /
> Cheers,
> Roman
> On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipingz...@gmail.com > <mailto:ipingz...@gmail.com>> wrote:
> Hi all:
> In the shell, I can use db.getCollectionNames() get a array of
> collection's name, but I don't know how to get collectionNames by
> c driver, can somebody help me ?
> Bruce Zhao
> 2012-10-08
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> <mailto:mongodb-user@googlegroups.com>
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> <mailto:mongodb-user%2Bunsubscribe@googlegroups.com>
> See also the IRC channel -- freenode.net#mongodb
> <http://freenode.net#mongodb>
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
you're welcome!
If you mind asking, are you located in or near Beijing?
I am asking, because I was asked by 10gen to organize the official Beijing
MongoDB office hours going to start in November. For now I am looking for
interested MongoDB users. :-)
> In short: there is no getCollectionName in the c drivers. Dunno if the
> drivers have been updated with this function yet. But as you can read in
> the post, it is quite easy to implement it at you own. :-)
> *"There is no getCollectionNames command. If you type
> db.getCollectionNames in the shell (note no parens) it will print out the
> source and you can see how it's implemented there."
> *
> Cheers,
> Roman
> On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipingz...@gmail.com> wrote:
>> Hi all:
>> In the shell, I can use db.getCollectionNames() get a array of
>> collection's name, but I don't know how to get collectionNames by c driver,
>> can somebody help me ?
>> Bruce Zhao
>> 2012-10-08
>> --
>> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongodb-user@googlegroups.com
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe@googlegroups.com
>> See also the IRC channel -- freenode.net#mongodb
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
-- Roman Alexis Anastasini
Senior Web Developer
CMUNE
On Monday, October 8, 2012 11:49:12 AM UTC+8, Roman Alexis Anastasini wrote:
> Hey Bruce,
> you're welcome!
> If you mind asking, are you located in or near Beijing?
> I am asking, because I was asked by 10gen to organize the official Beijing > MongoDB office hours going to start in November. For now I am looking for > interested MongoDB users. :-)
> Best,
> Roman
> On Mon, Oct 8, 2012 at 11:43 AM, Bruce Zhao <ipin...@gmail.com<javascript:>
> > wrote:
>> Roman,
>> Thank you very much!! It's helpful!!
>> 于 2012年10月08日 09:58, Roman Alexis Anastasini 写道:
>> In short: there is no getCollectionName in the c drivers. Dunno if the >> drivers have been updated with this function yet. But as you can read in >> the post, it is quite easy to implement it at you own. :-)
>> *"There is no getCollectionNames command. If you type >> db.getCollectionNames in the shell (note no parens) it will print out the >> source and you can see how it's implemented there."
>> *
>> Cheers,
>> Roman
>> On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipin...@gmail.com<javascript:>
>> > wrote:
>>> Hi all:
>>> In the shell, I can use db.getCollectionNames() get a array of >>> collection's name, but I don't know how to get collectionNames by c driver, >>> can somebody help me ?
>>> Bruce Zhao
>>> 2012-10-08
>>> -- >>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com<javascript:>
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@googlegroups.com <javascript:>
>>> See also the IRC channel -- freenode.net#mongodb
>> -- >> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com<javascript:>
>> To unsubscribe from this group, send email to
>> mongodb-user...@googlegroups.com <javascript:>
>> See also the IRC channel -- freenode.net#mongodb
>> -- >> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com<javascript:>
>> To unsubscribe from this group, send email to
>> mongodb-user...@googlegroups.com <javascript:>
>> See also the IRC channel -- freenode.net#mongodb
> -- > Roman Alexis Anastasini
> Senior Web Developer
> CMUNE
On Monday, October 8, 2012 11:54:00 AM UTC+8, Hanson Lu wrote:
> Is google groups bug? I start the thread, but your post seems not related.
> On Monday, October 8, 2012 11:49:12 AM UTC+8, Roman Alexis Anastasini > wrote:
>> Hey Bruce,
>> you're welcome!
>> If you mind asking, are you located in or near Beijing?
>> I am asking, because I was asked by 10gen to organize the official >> Beijing MongoDB office hours going to start in November. For now I am >> looking for interested MongoDB users. :-)
>> Best,
>> Roman
>> On Mon, Oct 8, 2012 at 11:43 AM, Bruce Zhao <ipin...@gmail.com> wrote:
>>> Roman,
>>> Thank you very much!! It's helpful!!
>>> 于 2012年10月08日 09:58, Roman Alexis Anastasini 写道:
>>> In short: there is no getCollectionName in the c drivers. Dunno if the >>> drivers have been updated with this function yet. But as you can read in >>> the post, it is quite easy to implement it at you own. :-)
>>> *"There is no getCollectionNames command. If you type >>> db.getCollectionNames in the shell (note no parens) it will print out the >>> source and you can see how it's implemented there."
>>> *
>>> Cheers,
>>> Roman
>>> On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipin...@gmail.com> wrote:
>>>> Hi all:
>>>> In the shell, I can use db.getCollectionNames() get a array of >>>> collection's name, but I don't know how to get collectionNames by c driver, >>>> can somebody help me ?
>>>> Bruce Zhao
>>>> 2012-10-08
>>>> -- >>>> You received this message because you are subscribed to the Google
>>>> Groups "mongodb-user" group.
>>>> To post to this group, send email to mongod...@googlegroups.com
>>>> To unsubscribe from this group, send email to
>>>> mongodb-user...@googlegroups.com
>>>> See also the IRC channel -- freenode.net#mongodb
>>> -- >>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@googlegroups.com
>>> See also the IRC channel -- freenode.net#mongodb
>>> -- >>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@googlegroups.com
>>> See also the IRC channel -- freenode.net#mongodb
>> -- >> Roman Alexis Anastasini
>> Senior Web Developer
>> CMUNE
1. You might try a few more threads to see if you keep getting more insert
performance.
2. That said, if your just counting inserts/second vs same with a single
table in an RDBMS I would not expect an advantage for mongo.
Performance-wise I'd expect the advantage for mongo in two places:
I) If the data requires child tables in the rdbms, for example orders
with order headers and lines in an rdbms that can be stored as one object
in mongo. This is quite common.
II) Built in sharding which makes it easy to scale out. Need to do a few
hundred thousand writes per second? No problem.
You might start a thread about tuning the insert with index and safe mode;
I'm sure folks smarter than me about that would be happy to look at
monvostat, iostat, settings etc and give some advice.
-- Max
On Oct 7, 2012 8:06 PM, "Hanson Lu" <hans...@gmail.com> wrote:
> Hi Osmar,
> I want to note that the inserting speed is seen from client side and
> server side(mongostat). The both is almost same.
> BTW, the result of insering with safe write conern , and no index.
> Thread average speed
> 5 4156
> 10 4898
> So if the speed with index and safe concern of my test is normal, is
> MongoDB not suitable for apps that require high inserting speed?
> It seems that there is no big improvement than Relational Db when
> inserting with index and safe conern.
> Hanson Lu
> On Saturday, September 29, 2012 4:08:39 PM UTC+8, Hanson Lu wrote:
>> I did some tests for insert performance of MongoDB. It shows insert
>> performance is slower when collection have index and SAFE write concern is
>> set.
>> Here is the test result
>> * Inserting with no index and safe write concern not set
>> threads count average speed
>> 1 10161
>> 2 17519
>> 3 15859
>> (thread count means the concurrent inserting thread count)
>> The speed seem is higher .
>> * Inserting with index and safe write concern not set
>> threads count average speed
>> 1 6320
>> 2 6240
>> * Inserting with index and safe write concern set
>> threads count average speed
>> 1 972
>> 2 1530
>> 5 2573
>> The data listed seems normal? the speed of inserting with index and
>> safe write concern, is only 1/5 of that with no index and no safe write
>> concern.
>> Is any way to speed up the performance for inserting with index and
>> safe write concern condition?
>> I only create two index for that collection.
>> Regards
>> Hanson
>> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
> you're welcome!
> If you mind asking, are you located in or near Beijing?
> I am asking, because I was asked by 10gen to organize the official > Beijing MongoDB office hours going to start in November. For now I am > looking for interested MongoDB users. :-)
> Best,
> Roman
> On Mon, Oct 8, 2012 at 11:43 AM, Bruce Zhao <ipingz...@gmail.com > <mailto:ipingz...@gmail.com>> wrote:
>> In short: there is no getCollectionName in the c drivers. Dunno
>> if the drivers have been updated with this function yet. But as
>> you can read in the post, it is quite easy to implement it at you
>> own. :-)
>> /"There is no getCollectionNames command. If you type
>> db.getCollectionNames in the shell (note no parens) it will print
>> out the source and you can see how it's implemented there."
>> /
>> Cheers,
>> Roman
>> On Mon, Oct 8, 2012 at 9:44 AM, Bruce Zhao <ipingz...@gmail.com
>> <mailto:ipingz...@gmail.com>> wrote:
>> Hi all:
>> In the shell, I can use db.getCollectionNames() get a
>> array of collection's name, but I don't know how to get
>> collectionNames by c driver, can somebody help me ?
>> Bruce Zhao
>> 2012-10-08
>> -- >> You received this message because you are subscribed to the
>> Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to
>> mongodb-user@googlegroups.com
>> <mailto:mongodb-user@googlegroups.com>
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe@googlegroups.com
>> <mailto:mongodb-user%2Bunsubscribe@googlegroups.com>
>> See also the IRC channel -- freenode.net#mongodb
>> <http://freenode.net#mongodb>
>> -- >> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to
>> mongodb-user@googlegroups.com <mailto:mongodb-user@googlegroups.com>
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe@googlegroups.com
>> <mailto:mongodb-user+unsubscribe@googlegroups.com>
>> See also the IRC channel -- freenode.net#mongodb
>> <http://freenode.net#mongodb>
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> <mailto:mongodb-user@googlegroups.com>
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> <mailto:mongodb-user%2Bunsubscribe@googlegroups.com>
> See also the IRC channel -- freenode.net#mongodb
> <http://freenode.net#mongodb>
> -- > Roman Alexis Anastasini
> Senior Web Developer
> CMUNE
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
On Monday, October 8, 2012 1:18:15 PM UTC+8, Max Schireson wrote: > A few comments:
> 1. You might try a few more threads to see if you keep getting more insert > performance.
> 2. That said, if your just counting inserts/second vs same with a single > table in an RDBMS I would not expect an advantage for mongo. > Performance-wise I'd expect the advantage for mongo in two places: > I) If the data requires child tables in the rdbms, for example orders > with order headers and lines in an rdbms that can be stored as one object > in mongo. This is quite common. > II) Built in sharding which makes it easy to scale out. Need to do a few > hundred thousand writes per second? No problem.
> You might start a thread about tuning the insert with index and safe mode; > I'm sure folks smarter than me about that would be happy to look at > monvostat, iostat, settings etc and give some advice.
> -- Max > On Oct 7, 2012 8:06 PM, "Hanson Lu" <han...@gmail.com <javascript:>> > wrote:
>> Hi Osmar, >> I want to note that the inserting speed is seen from client side and >> server side(mongostat). The both is almost same.
>> BTW, the result of insering with safe write conern , and no index. >> Thread average speed >> 5 4156 >> 10 4898
>> So if the speed with index and safe concern of my test is normal, is >> MongoDB not suitable for apps that require high inserting speed? >> It seems that there is no big improvement than Relational Db when >> inserting with index and safe conern.
>> Hanson Lu >> On Saturday, September 29, 2012 4:08:39 PM UTC+8, Hanson Lu wrote:
>>> I did some tests for insert performance of MongoDB. It shows insert >>> performance is slower when collection have index and SAFE write concern is >>> set.
>>> Here is the test result
>>> * Inserting with no index and safe write concern not set >>> threads count average speed >>> 1 10161 >>> 2 17519 >>> 3 15859 >>> (thread count means the concurrent inserting thread count) >>> The speed seem is higher . >>> * Inserting with index and safe write concern not set >>> threads count average speed >>> 1 6320 >>> 2 6240
>>> * Inserting with index and safe write concern set >>> threads count average speed >>> 1 972 >>> 2 1530 >>> 5 2573
>>> The data listed seems normal? the speed of inserting with index and >>> safe write concern, is only 1/5 of that with no index and no safe write >>> concern. >>> Is any way to speed up the performance for inserting with index and >>> safe write concern condition? >>> I only create two index for that collection.
>>> Regards >>> Hanson
>>> -- >> You received this message because you are subscribed to the Google >> Groups "mongodb-user" group. >> To post to this group, send email to mongod...@googlegroups.com<javascript:> >> To unsubscribe from this group, send email to >> mongodb-user...@googlegroups.com <javascript:> >> See also the IRC channel -- freenode.net#mongodb
> Hi Max
> I did test more threads, but it did not getting more insert
> performance, sometimes it get more worse.
> Hanson
> On Monday, October 8, 2012 1:18:15 PM UTC+8, Max Schireson wrote:
>> A few comments:
>> 1. You might try a few more threads to see if you keep getting more
>> insert performance.
>> 2. That said, if your just counting inserts/second vs same with a single
>> table in an RDBMS I would not expect an advantage for mongo.
>> Performance-wise I'd expect the advantage for mongo in two places:
>> I) If the data requires child tables in the rdbms, for example orders
>> with order headers and lines in an rdbms that can be stored as one object
>> in mongo. This is quite common.
>> II) Built in sharding which makes it easy to scale out. Need to do a
>> few hundred thousand writes per second? No problem.
>> You might start a thread about tuning the insert with index and safe
>> mode; I'm sure folks smarter than me about that would be happy to look at
>> monvostat, iostat, settings etc and give some advice.
>> -- Max
>> On Oct 7, 2012 8:06 PM, "Hanson Lu" <han...@gmail.com> wrote:
>>> Hi Osmar,
>>> I want to note that the inserting speed is seen from client side and
>>> server side(mongostat). The both is almost same.
>>> BTW, the result of insering with safe write conern , and no index.
>>> Thread average speed
>>> 5 4156
>>> 10 4898
>>> So if the speed with index and safe concern of my test is normal, is
>>> MongoDB not suitable for apps that require high inserting speed?
>>> It seems that there is no big improvement than Relational Db when
>>> inserting with index and safe conern.
>>> Hanson Lu
>>> On Saturday, September 29, 2012 4:08:39 PM UTC+8, Hanson Lu wrote:
>>>> I did some tests for insert performance of MongoDB. It shows insert
>>>> performance is slower when collection have index and SAFE write concern is
>>>> set.
>>>> Here is the test result
>>>> * Inserting with no index and safe write concern not set
>>>> threads count average speed
>>>> 1 10161
>>>> 2 17519
>>>> 3 15859
>>>> (thread count means the concurrent inserting thread count)
>>>> The speed seem is higher .
>>>> * Inserting with index and safe write concern not set
>>>> threads count average speed
>>>> 1 6320
>>>> 2 6240
>>>> * Inserting with index and safe write concern set
>>>> threads count average speed
>>>> 1 972
>>>> 2 1530
>>>> 5 2573
>>>> The data listed seems normal? the speed of inserting with index and
>>>> safe write concern, is only 1/5 of that with no index and no safe write
>>>> concern.
>>>> Is any way to speed up the performance for inserting with index and
>>>> safe write concern condition?
>>>> I only create two index for that collection.
>>>> Regards
>>>> Hanson
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@**googlegroups.com
>>> See also the IRC channel -- freenode.net#mongodb
>> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
On Monday, October 8, 2012 9:39:15 PM UTC-4, Hanson Lu wrote:
> I did test more threads, but it did not getting more insert > performance, sometimes it get more worse.
As I stated before the issue is you are blocking on each thread for a reply and also blocking other threads from using the connection to send more requests. If you profile the application you will see the application is always asleep waiting for a response or waiting to get a connection from the pool.
Note that the 10gen (legacy) driver's throughput (black bars) fairly quickly reaches a maximum for each of the connection counts. If you need more throughput you have to increase both the thread count _and_ the connection count using the 10gen provided drivers. If you have a small application deployment this scales fairly well. If you need this to scale across even a medium size cluster serving a number of clients the connection explosion can start to cause scheduler and other problems.
The other option is to find a driver that doesn't block the connection waiting for the response. The only production ready version I know of is the asynchronous Java driver<http://www.allanbank.com/mongodb-async-driver/>my team created. You can see in the chart above it scales fairly well[1] and tops out at ~13K operations/second with a "SAFE" write concern. Much better than the 2K you are seeing.
I encourage you to give it a try. Feel free to ask questions directly to me or on this group.
If you are not using Java you can try increasing the thread and connection count and see if that gets you where you need to be.
Rob.
[1] Remember this was a benchmark so the client threads are doing no real work so I would expect for a real workload the scaling to be even better.
On Tuesday, October 9, 2012 11:01:21 AM UTC+8, Rob Moore wrote:
> On Monday, October 8, 2012 9:39:15 PM UTC-4, Hanson Lu wrote:
>> I did test more threads, but it did not getting more insert >> performance, sometimes it get more worse.
> As I stated before the issue is you are blocking on each thread for a > reply and also blocking other threads from using the connection to send > more requests. If you profile the application you will see the application > is always asleep waiting for a response or waiting to get a connection from > the pool.
> Note that the 10gen (legacy) driver's throughput (black bars) fairly > quickly reaches a maximum for each of the connection counts. If you need > more throughput you have to increase both the thread count _and_ the > connection count using the 10gen provided drivers. If you have a small > application deployment this scales fairly well. If you need this to scale > across even a medium size cluster serving a number of clients the > connection explosion can start to cause scheduler and other problems.
> The other option is to find a driver that doesn't block the connection > waiting for the response. The only production ready version I know of is > the asynchronous Java driver<http://www.allanbank.com/mongodb-async-driver/>my team created. You can see in the chart above it scales fairly well[1] > and tops out at ~13K operations/second with a "SAFE" write concern. Much > better than the 2K you are seeing.
> I encourage you to give it a try. Feel free to ask questions directly to > me or on this group.
> If you are not using Java you can try increasing the thread and connection > count and see if that gets you where you need to be.
> Rob.
> [1] Remember this was a benchmark so the client threads are doing no real > work so I would expect for a real workload the scaling to be even better.
Hi Rob. I have tried with your driver, the most fast speed i saw is ~11k writes, with 200 threads. The strange thing is the performance is achieved when max connecttion count is set to 1(mongo.getConfig().setMaxConnectionCount(1)) if i change the max connection count to 1000, the performance is about ~6k, slower than max connection count 1.
On Tuesday, October 9, 2012 11:01:21 AM UTC+8, Rob Moore wrote:
> On Monday, October 8, 2012 9:39:15 PM UTC-4, Hanson Lu wrote:
>> I did test more threads, but it did not getting more insert >> performance, sometimes it get more worse.
> As I stated before the issue is you are blocking on each thread for a > reply and also blocking other threads from using the connection to send > more requests. If you profile the application you will see the application > is always asleep waiting for a response or waiting to get a connection from > the pool.
> Note that the 10gen (legacy) driver's throughput (black bars) fairly > quickly reaches a maximum for each of the connection counts. If you need > more throughput you have to increase both the thread count _and_ the > connection count using the 10gen provided drivers. If you have a small > application deployment this scales fairly well. If you need this to scale > across even a medium size cluster serving a number of clients the > connection explosion can start to cause scheduler and other problems.
> The other option is to find a driver that doesn't block the connection > waiting for the response. The only production ready version I know of is > the asynchronous Java driver<http://www.allanbank.com/mongodb-async-driver/>my team created. You can see in the chart above it scales fairly well[1] > and tops out at ~13K operations/second with a "SAFE" write concern. Much > better than the 2K you are seeing.
> I encourage you to give it a try. Feel free to ask questions directly to > me or on this group.
> If you are not using Java you can try increasing the thread and connection > count and see if that gets you where you need to be.
> Rob.
> [1] Remember this was a benchmark so the client threads are doing no real > work so I would expect for a real workload the scaling to be even better.
On Tuesday, October 9, 2012 8:44:59 AM UTC-4, Hanson Lu wrote:
> I have tried with your driver, the most fast speed i saw is ~11k writes, > with 200 threads. > The strange thing is the performance is achieved when max connecttion > count is set to 1(mongo.getConfig().setMaxConnectionCount(1)) > if i change the max connection count to 1000, the performance is about > ~6k, slower than max connection count 1.
Hanson,
Without knowing the specifics I'm not surprised. 1000 is a lot of connections. Part of the goal with the driver was to keep the connection count low[1]. When sending a request it does a linear search for an idle connection to try and spread the requests as evenly as possible. That search, with 1000 connections, is probably causing the slow down. Sitting here, there are probably some things that can be done to reduce the over head. I'll get the team to look into some of those.
Suggestion to get the very best throughput:
- For a application that is not doing much processing and is feeding the driver to capacity I would keep the thread to connection ratio around 10. e.g, 20 threads ==> 2 connections. - Cut the number of threads down, if possible. The driver ties to keep the running threads running and not blocked. If it works then the extra thread contention is hurting performance. - If you are willing to trade some CPU for shorter latency switch the lock type: - mongo.getConfig().setLockType(LockType<http://www.allanbank.com/mongodb-async-driver/apidocs/index.html?com/...>.LOW_LATENCY_SPIN);
- This must be done before a connection is created to MongoDB (query, command, list, etc.), only connections created after setting the lock type will use the new lock type.
Your now able to push 11K inserts instead of 2.5K. 11K is in the ballpark of what you saw with the NONE write concern and no index using the legacy driver. What is your goal? Can you provide some details on what you are actually doing so I can give more concrete suggestions? Are you only ever going to insert documents?
Rob.
[1] My personal goal is to saturate a mongod instance with a single connection. No one here thinks we can do it. I like a good challenge.
On Wednesday, October 10, 2012 8:38:28 AM UTC+8, Rob Moore wrote:
> On Tuesday, October 9, 2012 8:44:59 AM UTC-4, Hanson Lu wrote:
>> I have tried with your driver, the most fast speed i saw is ~11k >> writes, with 200 threads. >> The strange thing is the performance is achieved when max connecttion >> count is set to 1(mongo.getConfig().setMaxConnectionCount(1)) >> if i change the max connection count to 1000, the performance is >> about ~6k, slower than max connection count 1.
> Hanson,
> Without knowing the specifics I'm not surprised. 1000 is a lot of > connections. Part of the goal with the driver was to keep the connection > count low[1]. When sending a request it does a linear search for an idle > connection to try and spread the requests as evenly as possible. That > search, with 1000 connections, is probably causing the slow down. Sitting > here, there are probably some things that can be done to reduce the over > head. I'll get the team to look into some of those.
> Suggestion to get the very best throughput:
> - For a application that is not doing much processing and is feeding > the driver to capacity I would keep the thread to connection ratio around > 10. e.g, 20 threads ==> 2 connections. > - Cut the number of threads down, if possible. The driver ties to > keep the running threads running and not blocked. If it works then the > extra thread contention is hurting performance. > - If you are willing to trade some CPU for shorter latency switch the > lock type: > - mongo.getConfig().setLockType(LockType<http://www.allanbank.com/mongodb-async-driver/apidocs/index.html?com/...>.LOW_LATENCY_SPIN);
> - This must be done before a connection is created to MongoDB > (query, command, list, etc.), only connections created after setting the > lock type will use the new lock type.
> Your now able to push 11K inserts instead of 2.5K. 11K is in the ballpark > of what you saw with the NONE write concern and no index using the legacy > driver. What is your goal? Can you provide some details on what you are > actually doing so I can give more concrete suggestions? Are you only ever > going to insert documents?
> Rob.
> [1] My personal goal is to saturate a mongod instance with a single > connection. No one here thinks we can do it. I like a good challenge.
Hi
Due to mongodb still use per thread per request model so it's better to use connections sparingly. I put a proxy based on twisted in front of my mongodb instances. All operations go to proxy firstly and will be dispatched smoothly. Because of the async nature of twisted, I don't need too much connections (I use 10 or 30).
I hope one day mongodb team may implement async model.
On Wednesday, October 10, 2012 8:38:28 AM UTC+8, Rob Moore wrote:
> On Tuesday, October 9, 2012 8:44:59 AM UTC-4, Hanson Lu wrote:
>> I have tried with your driver, the most fast speed i saw is ~11k >> writes, with 200 threads. >> The strange thing is the performance is achieved when max connecttion >> count is set to 1(mongo.getConfig().setMaxConnectionCount(1)) >> if i change the max connection count to 1000, the performance is >> about ~6k, slower than max connection count 1.
> Hanson,
> Without knowing the specifics I'm not surprised. 1000 is a lot of > connections. Part of the goal with the driver was to keep the connection > count low[1]. When sending a request it does a linear search for an idle > connection to try and spread the requests as evenly as possible. That > search, with 1000 connections, is probably causing the slow down. Sitting > here, there are probably some things that can be done to reduce the over > head. I'll get the team to look into some of those.
> Suggestion to get the very best throughput:
> - For a application that is not doing much processing and is feeding > the driver to capacity I would keep the thread to connection ratio around > 10. e.g, 20 threads ==> 2 connections. > - Cut the number of threads down, if possible. The driver ties to > keep the running threads running and not blocked. If it works then the > extra thread contention is hurting performance. > - If you are willing to trade some CPU for shorter latency switch the > lock type: > - mongo.getConfig().setLockType(LockType<http://www.allanbank.com/mongodb-async-driver/apidocs/index.html?com/...>.LOW_LATENCY_SPIN);
> - This must be done before a connection is created to MongoDB > (query, command, list, etc.), only connections created after setting the > lock type will use the new lock type.
> Your now able to push 11K inserts instead of 2.5K. 11K is in the ballpark > of what you saw with the NONE write concern and no index using the legacy > driver. What is your goal? Can you provide some details on what you are > actually doing so I can give more concrete suggestions? Are you only ever > going to insert documents?
> Rob.
> [1] My personal goal is to saturate a mongod instance with a single > connection. No one here thinks we can do it. I like a good challenge.
ops/sec is very dependent on your hardware, network, and disk. I wouldn't be able to tell you if that is normal or not as the numbers by themselves don't really mean anything with the context of your machine. Though I can tell you the general trend you are seeing with increasing threads seems correct.
On Wednesday, October 10, 2012 5:07:40 AM UTC-4, Hanson Lu wrote:
> Hi Rob
> I just want to know the max insert performance of mongodb for my app.
> I made a test with 2 connection, different threads, the result is > Thr Insert ops/per seconds > 1 ~400 > 2 ~700 > 5 ~2k > 10 ~5k > 20 ~8k > 40 ~9k
> The single thread performance is 400 ops/sec, is it normal. > Hanson
> On Wednesday, October 10, 2012 8:38:28 AM UTC+8, Rob Moore wrote:
>> On Tuesday, October 9, 2012 8:44:59 AM UTC-4, Hanson Lu wrote:
>>> I have tried with your driver, the most fast speed i saw is ~11k >>> writes, with 200 threads. >>> The strange thing is the performance is achieved when max connecttion >>> count is set to 1(mongo.getConfig().setMaxConnectionCount(1)) >>> if i change the max connection count to 1000, the performance is >>> about ~6k, slower than max connection count 1.
>> Hanson,
>> Without knowing the specifics I'm not surprised. 1000 is a lot of >> connections. Part of the goal with the driver was to keep the connection >> count low[1]. When sending a request it does a linear search for an idle >> connection to try and spread the requests as evenly as possible. That >> search, with 1000 connections, is probably causing the slow down. Sitting >> here, there are probably some things that can be done to reduce the over >> head. I'll get the team to look into some of those.
>> Suggestion to get the very best throughput:
>> - For a application that is not doing much processing and is feeding >> the driver to capacity I would keep the thread to connection ratio around >> 10. e.g, 20 threads ==> 2 connections. >> - Cut the number of threads down, if possible. The driver ties to >> keep the running threads running and not blocked. If it works then the >> extra thread contention is hurting performance. >> - If you are willing to trade some CPU for shorter latency switch the >> lock type: >> - mongo.getConfig().setLockType(LockType<http://www.allanbank.com/mongodb-async-driver/apidocs/index.html?com/...>.LOW_LATENCY_SPIN);
>> - This must be done before a connection is created to MongoDB >> (query, command, list, etc.), only connections created after setting the >> lock type will use the new lock type.
>> Your now able to push 11K inserts instead of 2.5K. 11K is in the >> ballpark of what you saw with the NONE write concern and no index using the >> legacy driver. What is your goal? Can you provide some details on what you >> are actually doing so I can give more concrete suggestions? Are you only >> ever going to insert documents?
>> Rob.
>> [1] My personal goal is to saturate a mongod instance with a single >> connection. No one here thinks we can do it. I like a good challenge.
