mongo query performance with concurrent operations

[Stone Fang]

Environment

a mongo sharding cluster environment 2*3(3replica set,each has two nodes.),1 config server,1 query router. each node is 4 cpu,8G Mem.

have a collection with just 20,000 documents. have a basic api to query the document.no sharding no indexing. the api only contains a db query.

Operation

1.query on mongo shell with explain("executionStats"),need about 40ms

2.with 1000/min send throughput(test with jmeter) it is okay,the average latency is about 60ms

3.with just 6000/min send throughput. got a terrible performance with average 1000ms. cannot believe this!

4.use mongotop to check,nearly 4s

test.course total(4633ms) read(4633ms) write(0ms)

since mongotop shows how much time was spend reading or writing each collection over the last second

this indicate that there are some slow queries.

5.use profile to check query

db.system.profile.find({ns:"test.course",millis:{$gt:200}}).limit(10).sort({ts:-1}).pretty()

there are no result returned.only one record great than 200ms

How to explain this,and how to solve this. also did another test. same collection structure.same api,the only difference is do a sharding on the collection.but the query didn't use shard key. got a vary different result,as it works with a normal latency,about 50ms. even with 60000/min send throughput.

Can anyone help to explain this,or any idea on further investigation

[Rhys Campbell]

Post an example of the query, a sample document, and the explain for the query.

You state "no indexing" which makes me think. What is the output of db.course.getIndexes()?

As an aside you have a bit of an unusual setup there. You might want to look at the MongoDB docs for advice... https://docs.mongodb.com/manual/core/sharded-cluster-components/#production-configuration

[Stone Fang]

"no indexing" means that no other index on any field except _id

[Stone Fang]

and don't think there is any problem with mongo cluster architecture.as one config server,one query router is ok.

And did a test as I mentioned,just enable sharding on the collection,it works.

[Rhys Campbell]

It might work but it's not standard; 2 node replset for example.

You'll still need to supply the other info asked for above if you want advice.