MongoDB concurrency problem

[JunKai Wu]

Hi all~

I have a testing for mongodb limitation.

Here is my architecture:

as the picture, I have a node1 with 3 instances and become a shard

a config db on node2 and 3 mongos on node 3~5

*Each nodes are ubuntu12.04-server with "ulimit -f unlimited -t unlimited -m unlimited -n 64000 -u 64000" command.

===================================================================

Testing :

Apps is written by java with mongo java driver 2.12.3.

Each app will generate 1,000 thread per second for inserting a record which is almost 427 bytes.(Write concern level : SAFE)

*That means, there are 3,000 threads concurrent write into shard1 per second.

===================================================================

Result : 

I run the testing for 3 minutes, means there should be 540,000 records in my database.

But I found some exception with : 

com.mongodb.MongoException$Network: Operation on server 127.0.0.1:30000 failed
        at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:217)
        at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:491)
        at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:430)
        at com.mongodb.DBCollectionImpl.insertWithCommandProtocol(DBCollectionImpl.java:387)
        at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:186)
        at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:165)
        at com.mongodb.DBCollection.insert(DBCollection.java:161)
        at com.mongodb.DBCollection.insert(DBCollection.java:107)
        at org.jongo.Insert.insert(Insert.java:67)
        at org.jongo.MongoCollection.insert(MongoCollection.java:144)
        at org.jongo.MongoCollection.insert(MongoCollection.java:136)
        at database.testing.operation.WriteDataThread.call(WriteDataThread.java:231)
        at database.testing.operation.WriteDataThread.call(WriteDataThread.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:196)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at org.bson.io.Bits.readFully(Bits.java:48)
        at org.bson.io.Bits.readFully(Bits.java:35)
        at org.bson.io.Bits.readFully(Bits.java:30)
        at com.mongodb.Response.<init>(Response.java:42)
        at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:494)
        at com.mongodb.DBCollectionImpl$3.execute(DBCollectionImpl.java:491)
        at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:210)
        ... 14 more

 ===================================================================

Problem : 

How can I tuning my shard1 for more concurrent access ? (mongodb tuning and OS tuning)

[Will Berkeley]

Could you clarify the setup a bit? 

1. You have 3 mongos and 1 config server (should really have 3) to support a cluster, but only one shard? Why are you using sharding if there's only one shard? 

2. It seems like all of your replica set members are on the same machine, which nullifies the point of having a replica set. The reasonable setup for this situation would be a single replica set with 3 members, on separate nodes, with no sharding components necessary.

3. What does it mean that the primary instance is "in-memory db"?

4. When you say the app generates 1000 threads per second, do you really mean you spawn a separate thread per request as opposed to having a stable set of 1000 threads, each of which generates 1 request per second?

5. Is there any error logged in the mongos or mongod logs? That would be more helpful than the Java socket exception.

For general information on optimization, check out the production notes.

-Will

[JunKai Wu]

Thanks for your replying

1. I build only one shard since it is not the bottleneck I faced in the previous test. The bottleneck I faced is that the application node cannot afford more than 1,000 concurrent threads per second. Thus I try to perform the test simutaneously with 3 application nodes (and mongos) and want to make the shard busier.

2. 3. I mount a memory block as a disk space and the dbpath of primary instance is located to the mounted memory disk. "In-memory" database is that all database data was stored in memory(Though all data will disappear when machine shutdown). For preventing machine shutdown situation, I build the replica set in one machine that primary instance(in-memory) could sync all data from memory to disk.

4. Yes, the app stably generates 1000 separate threads per second, each thread will request mongo db an insert command.

5. I didn't save the log before, I will supplement logged after I re-test the case.

Thank you.