DataFileSync flushing mmaps takes 60 seconds, hangs server

[Marc Ilgen]

We are running MongoDB 2.6.9 on a 64 bit Ubuntu Linux VM. When I look in the database folder, this is what I see with ls -lh:
-rw------- 1 mongodb mongodb  64M Apr 15 14:26 admin.0
-rw------- 1 mongodb mongodb  16M Apr 15 14:26 admin.ns
-rw------- 1 mongodb nogroup  64M Jun 17 11:56 my_database.0
-rw------- 1 mongodb nogroup 128M May  7 23:15 my_database.1
-rw------- 1 mongodb nogroup 256M Jun 15 07:40 my_database.2
-rw------- 1 mongodb nogroup 512M May 14 07:26 my_database.3
-rw------- 1 mongodb nogroup 1.0G Jun 10 09:26 my_database.4
-rw------- 1 mongodb nogroup 2.0G Jun  2 11:25 my_database.5
-rw------- 1 mongodb nogroup 2.0G Jun 17 12:03 my_database.6
-rw------- 1 mongodb nogroup 2.0G Jun 17 12:03 my_database.7
-rw------- 1 mongodb nogroup 2.0G Jun 17 12:03 my_database.8
-rw------- 1 mongodb nogroup  16M Jun 17 12:03 my_database.ns
drwxr-xr-x 2 mongodb mongodb 4.0K Jun 16 13:16 journal
-rw------- 1 mongodb mongodb  64M Jun 16 13:16 local.0
-rw------- 1 mongodb mongodb  16M Jun 16 13:16 local.ns
-rwxr-xr-x 1 mongodb mongodb    5 Jun 16 13:16 mongod.lock

So we have a few 2GB files. But the problem is that periodically we get a log message like this:
2015-06-12T18:10:01.790-0700 [DataFileSync] flushing mmaps took 62192ms  for 14 files

During this period of time mongodb locks up and all queries fail.

How can we fix this issue?

[Allan Bazinet]

Typically that message means that the I/O subsystem on which your data volume resides can't keep up with background flush activity.  What is the I/O activity on that volume?