Mongo not utilising all CPUs

[ndss]

Hi,

I have a large (9 nodes) mongoDB cluster running on WiredTiger 3.0.11 with snappy compression.

Database size is < 1 gig.  Fits in RAM easily (each host has 32 gig)

Hosts are running on c4.2xlarge AWS EC2 instances -> each host has 8 CPUs.  

Each host services approximately 200 requests per second.

The general query profile is targeted queries with a large set of parameters being passed to find a small range of specific records. 

Each data request is quite different than the previous data request.  

(Probably not an ideal use case for mongo).

It's a read heavy database.

Mongostat output during an offpeak time:

AR| AW

 1|0  

 2|0  

 1|0  

 1|0  

 1|0  

 5|0  

 1|0  

 2|0  

 5|0  

 2|0  

db.serverStatus().connections

{

   "current" : 380,

   "available" : 52048,

   "totalCreated" : NumberLong(1998452)

}

By all accounts the cluster is extremely overprovisioned but users are still seeing intermittent mongo timeouts, and also seeing application response time decrease as traffic is ramped up.

When running top on the all hosts I can see that only one CPU of the 8 available is being utilised.  % CPU will max out at 100% of one CPU but never goes over this number (I have a CPU utilisation graph going back for a 3 month period that shows the total user CPU as being never above 10%, 1/8ths of the CPUs on the host is 12.5%).   The load average never goes above 1 (also shown clearly on a graph).

From my research I can see that wiredTiger is meant to take advantage of multiple cores for reads.   

Can somebody please help me understand:

a)  Why Mongo isn't utilising more than one CPU when it's maxing out one CPU and struggling to service requests?

b)  Are there any ways (possibly at the application architecture level) to force it to utilise more CPUs?

Adding in more hosts does help with the performance issues in the short term.

This usecase is probably a good candidate for sharding but it seems a little overkill to shard a 1 gig database.

Any help would be appreciated.

Thanks.

[Kevin Adistambha]

Hi,

Could you provide more details regarding your deployment, such as:

• Your O/S version
• Your filesystem (e.g. XFS or ext4) and their mount options
• Your topology (replica set or sharded cluster, and more details regarding each node)
• Are all 9 nodes running on a single c4 instance, or do you have 9 separate c4 instances?
• Are the storage using Provisioned IOPS EBS volumes?
• Do you have other processes besides mongod running on each host (e.g. another mongod, web server, application server, etc.)
• Are there any errors/warnings in the logs, either during the startup of each node, or during the stalls?
• Are there any mongod settings that were set differently from the default, e.g. WiredTiger cache setting, etc? It will be helpful if you can post the output of db.serverCmdLineOpts() or the contents of any config files.
• If this is a 9-node replica set setup, are you using Secondary read preference in your application?

Please also ensure that you have setup your deployment according to the recommendations outlined in the Production Notes and the Operations Checklist. Also, the Amazon EC2 manual page may be relevant for recommendations specific to AWS.

The general query profile is targeted queries with a large set of parameters being passed to find a small range of specific records.

To understand your use case, could you provide:

• Some example documents
• Some example queries
• The explain results of a query

Adding in more hosts does help with the performance issues in the short term.

If your setup is a 9-node replica set, did you mean adding new Secondaries?

To answer your specific questions:

a) Why Mongo isn’t utilising more than one CPU when it’s maxing out one CPU and struggling to service requests?

Typically, MongoDB is less CPU-intensive and more disk-intensive. The issue you’re describing sounds like MongoDB is blocked on other things besides CPU, with disk being the most likely cause.

b) Are there any ways (possibly at the application architecture level) to force it to utilise more CPUs?

Currently there’s no method available in MongoDB to force it to use more CPU.

Best regards,
Kevin

​