mongo 3.4.4 serverStatus was very slow

[Need Help]

Our mongo server on the cloud has become very slow lately. We have no clue as what has happened. Can anyone help, please?
-----------------------------------------------------------------------------------
2018-05-15T07:09:43.420-0700 I COMMAND [ftdc] serverStatus was very slow: { after basic: 209, after asserts: 229, after backgroundFlushing: 330, after connections: 572, after dur: 676, after extra_info: 872, after globalLock: 1056, after locks: 1558, after network: 1776, after opLatencies: 1967, after opcounters: 2131, after opcountersRepl: 2253, after repl: 2488, after security: 2696, after storageEngine: 3072, after tcmalloc: 4119, after wiredTiger: 33447, at end: 36557 } 2018-05-15T07:09:45.090-0700 I COMMAND [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 112ms 2018-05-15T07:09:45.730-0700 I COMMAND [conn284621] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1455ms 2018-05-15T07:09:49.123-0700 I COMMAND [conn284627] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1159ms 2018-05-15T07:09:49.123-0700 I COMMAND [conn284624] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1017ms 2018-05-15T07:09:49.334-0700 I - [conn284640] end connection 10.0.1.128:34630 (33 connections now open) 2018-05-15T07:09:49.752-0700 I - [conn284641] end connection 10.0.1.128:34642 (33 connections now open) 2018-05-15T07:09:49.977-0700 I COMMAND [conn284637] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1321ms 2018-05-15T07:09:51.683-0700 I COMMAND [conn284635] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1547ms 2018-05-15T07:09:51.683-0700 I COMMAND [conn284633] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1543ms 2018-05-15T07:09:51.683-0700 I COMMAND [conn284632] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1547ms 2018-05-15T07:09:51.683-0700 I COMMAND [conn284639] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 1547ms 2018-05-15T07:09:54.114-0700 I COMMAND [conn213025] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 504ms 2018-05-15T07:09:54.281-0700 I COMMAND [conn284621] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 514ms 2018-05-15T07:09:57.404-0700 I COMMAND [conn233822] command admin.$cmd command: isMaster { ismaster: 1 } numYields:0 reslen:189 locks:{} protocol:op_query 670ms 2018-05-15T07:09:57.882-0700 I COMMAND [ftdc] serverStatus was very slow: { after basic: 236, after asserts: 535, after backgroundFlushing: 704, after connections: 1020, after dur: 1242, after extra_info: 1626, after globalLock: 2076, after locks: 3782, after network: 3843, after opLatencies: 3994, after opcounters: 4124, after opcountersRepl: 4185, after repl: 4359, after security: 4471, after storageEngine: 4596, after tcmalloc: 4785, after wiredTiger: 7458, at end: 8291 }

[Kevin Adistambha]

Hi

Our mongo server on the cloud has become very slow lately

Typically the serverStatus was very slow message is printed when the server is being overwhelmed with incoming request, or it needs to perform a lot more work using an underprovisioned hardware.

Could you specify which cloud you have your MongoDB server on? Also, it will be helpful if you provide more information on how the mongod server is provisioned (e.g. RAM & CPU, the MongoDB and the OS version, the topology — standalone, replica set, or sharded cluster) and what operation are you doing when this was printed in the logs.

Best regards
Kevin

​

[Need Help]

Hello:

Thanks for the reply!

Here are some information:
------------------------------------------------------------------

The cloud is Amazon AWS. 

The mongo server hardware is r4.2xlarge. It has 8 vCPU, 61 GB memory, running Ubuntu 14.04 os. It's a standalone mongo server.

For the operation, it's mainly writing to collection records.

Thanks!

[Kevin Adistambha]

Hi

Is the machine dedicated to just running a single mongod, or are you running some other processes there as well? E.g., web server, application server, etc.?

From the MongoDB side, is there anything that are changed from the defaults, e.g. what is the output of db.serverCmdLineOpts() in the mongo shell and any config file content? What is your full MongoDB version (e.g. 3.6.4)?

In terms of the provisioned hardware, are you using SSD or a spinning disk?

About the operation, how exactly are you writing the data? Are you doing any parallel writes e.g. by using multiple threads, multiple apps, etc.?

There may be some relevant information in the output of mongostat, mongotop, and iostat. What are the output of these programs show during the slowdown?

Best regards
Kevin

​