replica set high disk io
435 views
Skip to first unread message
bruceguodong
Apr 14, 2014, 5:41:02 AM4/14/14
to mongod...@googlegroups.com
Hi, all:
I have recently migerated data from mysql to mongodb, but found the high disk i/o problem after the deployment.
We choose the simplest configuration which only contains two nodes, one serve as primary, and the other as secondary and arbiter.
Our document increment is about 20,000,000 daily. After the migration, the query performance is getting much better obviously,
but the high disk io problem confused me somehow.
Disk I/O (monitored by Nagios):
169(primary):
tps: 138(avg) 1115.69(max)
KB_read: 4.39(avg) 1184(max)
KB_write: 1159(avg) 9304(max)
170(secondary, arbiter):
tps: 238(avg) 750(max)
KB_read: 8(avg) 780(max)
KB_write: 1831(avg) 5734(max)
Have someone met the same problem, and can you give me any suggestion for handling this.
By the way, our db servers's disk capacity is 5TB, the oplog's value as the default setting(250G more or less).
memory total is 64GB, the number of cpu's processor is 24.
So, has any one know the reason or can give some suggestion, thanks a lot.
I have recently migerated data from mysql to mongodb, but found the high disk i/o problem after the deployment.
We choose the simplest configuration which only contains two nodes, one serve as primary, and the other as secondary and arbiter.
Our document increment is about 20,000,000 daily. After the migration, the query performance is getting much better obviously,
but the high disk io problem confused me somehow.
Disk I/O (monitored by Nagios):
169(primary):
tps: 138(avg) 1115.69(max)
KB_read: 4.39(avg) 1184(max)
KB_write: 1159(avg) 9304(max)
170(secondary, arbiter):
tps: 238(avg) 750(max)
KB_read: 8(avg) 780(max)
KB_write: 1831(avg) 5734(max)
Have someone met the same problem, and can you give me any suggestion for handling this.
By the way, our db servers's disk capacity is 5TB, the oplog's value as the default setting(250G more or less).
memory total is 64GB, the number of cpu's processor is 24.
So, has any one know the reason or can give some suggestion, thanks a lot.
William Zola
Apr 14, 2014, 12:25:05 PM4/14/14
to mongod...@googlegroups.com
Hi Bruce!
High disk IOPS can be due to many different causes, either individually or in combination. Common causes include:
Missing indexes, or other schema design problems
Working set exceeds available RAM
Initial start up (loading data and indexes with a “cold” cache)
Inefficient pre-allocation due to using a filesystem other than ext4 or xfs
Incorrect disk subsystem configuration
Other processes running on the host
Hard disk degradation or failure
The most likely cause is that you have inefficient queries in your system: you can start by looking in your mongodb.log file from the primary and checking for slow-running queries. If you have any operations where n is much smaller than nscanned, those are good candidates for causing the excessive I/O.
-William
bruceguodong
Apr 15, 2014, 4:33:24 AM4/15/14
to mongod...@googlegroups.com
Hi, William:
Thanks for your reply.
I have read the official manual for MongoDB but do not find any reason that may lead to the problem.
Our MongoDB's version is 2.4.9.
And our histroy data is also stored in mongodb that mostly for querying.
the mongostat's output as below (secondary server):
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn set repl time
*110 8 *154 *0 0 26|0 0 286g 574g 62.1g 0 reportdb:2.1% 0 0|0 0|0 7k 9k 40 QbSet0 SEC 16:16:18
*438 4 *650 *0 0 14|0 0 286g 574g 62.1g 1 reportdb:8.0% 0 0|0 0|0 3k 6k 39 QbSet0 SEC 16:16:19
*181 11 *213 *0 0 18|0 0 286g 574g 62.1g 0 reportdb:3.1% 0 0|0 0|0 6k 8k 40 QbSet0 SEC 16:16:20
*220 7 *252 *0 0 15|0 0 286g 574g 62.1g 0 reportdb:3.6% 0 0|0 0|0 5k 7k 39 QbSet0 SEC 16:16:21
*247 6 *336 *0 0 21|0 0 286g 574g 62.1g 0 reportdb:4.1% 0 0|0 0|0 5k 8k 40 QbSet0 SEC 16:16:22
*266 6 *439 *0 0 13|0 0 286g 574g 62.1g 1 reportdb:4.5% 0 0|0 0|0 3k 6k 39 QbSet0 SEC 16:16:23
*0 10 *0 *0 0 25|0 0 286g 574g 62.1g 0 reportdb:0.2% 0 0|0 0|0 8k 8k 39 QbSet0 SEC 16:16:24
*386 7 2|530 *0 0 31|0 0 286g 574g 62.1g 0 reportdb:6.6% 0 0|0 0|0 6k 22k 39 QbSet0 SEC 16:16:25
*202 10 3|336 *0 0 37|0 0 286g 574g 62.1g 0 reportdb:3.6% 0 0|0 0|1 7k 24k 41 QbSet0 SEC 16:16:26
*293 8 *466 *0 0 19|0 0 286g 574g 62.1g 0 reportdb:5.8% 0 0|0 0|0 6k 10k 39 QbSet0 SEC 16:16:27
a. the mapped file is 286g, vsize 574g, but our RAM is 64g.
b. our most frequently operation is upsert and query.
the query performance is so good(just costs 3~4 msec), but both the indexes and the last flush time items are getting worse now.
the item a and b I have mentioned above may be the reason that cause the problem or not ?
在 2014年4月15日星期二UTC+8上午12时25分05秒，William Zola写道：
Thanks for your reply.
I have read the official manual for MongoDB but do not find any reason that may lead to the problem.
Our MongoDB's version is 2.4.9.
And our histroy data is also stored in mongodb that mostly for querying.
the mongostat's output as below (secondary server):
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn set repl time
*110 8 *154 *0 0 26|0 0 286g 574g 62.1g 0 reportdb:2.1% 0 0|0 0|0 7k 9k 40 QbSet0 SEC 16:16:18
*438 4 *650 *0 0 14|0 0 286g 574g 62.1g 1 reportdb:8.0% 0 0|0 0|0 3k 6k 39 QbSet0 SEC 16:16:19
*181 11 *213 *0 0 18|0 0 286g 574g 62.1g 0 reportdb:3.1% 0 0|0 0|0 6k 8k 40 QbSet0 SEC 16:16:20
*220 7 *252 *0 0 15|0 0 286g 574g 62.1g 0 reportdb:3.6% 0 0|0 0|0 5k 7k 39 QbSet0 SEC 16:16:21
*247 6 *336 *0 0 21|0 0 286g 574g 62.1g 0 reportdb:4.1% 0 0|0 0|0 5k 8k 40 QbSet0 SEC 16:16:22
*266 6 *439 *0 0 13|0 0 286g 574g 62.1g 1 reportdb:4.5% 0 0|0 0|0 3k 6k 39 QbSet0 SEC 16:16:23
*0 10 *0 *0 0 25|0 0 286g 574g 62.1g 0 reportdb:0.2% 0 0|0 0|0 8k 8k 39 QbSet0 SEC 16:16:24
*386 7 2|530 *0 0 31|0 0 286g 574g 62.1g 0 reportdb:6.6% 0 0|0 0|0 6k 22k 39 QbSet0 SEC 16:16:25
*202 10 3|336 *0 0 37|0 0 286g 574g 62.1g 0 reportdb:3.6% 0 0|0 0|1 7k 24k 41 QbSet0 SEC 16:16:26
*293 8 *466 *0 0 19|0 0 286g 574g 62.1g 0 reportdb:5.8% 0 0|0 0|0 6k 10k 39 QbSet0 SEC 16:16:27
a. the mapped file is 286g, vsize 574g, but our RAM is 64g.
b. our most frequently operation is upsert and query.
the query performance is so good(just costs 3~4 msec), but both the indexes and the last flush time items are getting worse now.
the item a and b I have mentioned above may be the reason that cause the problem or not ?
在 2014年4月15日星期二UTC+8上午12时25分05秒，William Zola写道：
William Zola
Apr 15, 2014, 8:02:30 AM4/15/14
to mongod...@googlegroups.com
Hi Bruce!
The official manual for MongoDB doesn't contain full instructions on how to diagnose high IOPS. The good news is that, from looking at your 'mongostat' output, your system does not appear to be in distress.
If you like, I can walk you through diagnosing your particular problem. Your particular problem is not obvious: I will need a *lot* of additional information from you in order to diagnose what's going on. Feel free to call a halt to the diagnostic process at any time.
Please be aware that MongoDB delegates much of the I/O management to the OS. Therefore, we'll need to gather a lot of OS-level information in order to do the diagnosis.
1) Let's start with the basics, and get a description of your OS and OS configuration. I'm going to assume that you're running on Linux since you mentioned Nagios and since your 'mongostat' output shows 0 page faults. (Due to SERVER-5799 I've never seen a Windows machine report 0 page faults.)
So, given that assumption, please let me know the following:
- What distribution and version of Linux are you running?
- Are you running on a physical or a virtual machine?
- If you're running on a virtual machine:
- What virtualization software are you using?
- Have you over-allocated RAM or CPU to this VM?
- Are you using memory ballooning?
- Are you using physical disks or network disks? If network disks, are you using a SAN or a NAS? If network disks, what network protocol are you using?
2) Next, please take one of your 'mongod.log' files, covering a time where you're experiencing high IOPS. Please make sure that it doesn't contain more than 24 hours worth of entries. You may need to make a copy of the log file and edit it to cut down the amount of data it has. Analyze it using the following command:
$ grep -v -E 'LockPinger|admin.\$cmd' mongod.log | \
awk '/[0-9]{3,}ms$/ {print $NF "\t" $0}' | \
sort -rn | head -200 > slow_operations.txtReplace "mongod.log" with the name of the log file you're analyzing, and make sure that the backslash ('\') is the last character on the line, with no trailing whitespace. Once this has run, examine the generated "slow_operations.txt' file. It will contain a sorted collection of log file entries, from the slowest to the fastest. Please let me know:
- How many entries there are for query, insert, and update (for example: grep update slow_operations.txt | wc -l)
- The time in milliseconds of the longest (first) entry
- Find the first entry which is a query or update, and let me know the values for 'n' and 'nscanned'
If there's no data for any of these, let me know that, too.
3) In addition, please post the output of the following commands:
$ df -h
$ mount -l
$ du -sh $DATADIR
$ ls -l $DATADIR
$ sudo blockdev --report
$ sudo cat /proc/mdstat
$ sudo lsblk -a
$ mongo --eval 'printjson(db.serverStatus({workingSet:1}).workingSet)'
$ mongo --eval 'printjson(db.serverStatus().mem)'
$ top -b -n 2
$ iostat -xtm 1 3(Replace $DATADIR with the full path of the --datadir of the mongoDB instance.)
Note that it's best to run the 'top' and 'iostat' commands at a time when Nagios is reporting high disk I/O.
4) In your original question, you say:
both the indexes and the last flush time items are getting worse now.
What do you mean by "The indexes are getting worse"? Where and how are you measuring this? What values are you seeing? What values are you seeing for last flush times?
5) Finally, if this system is in MMS, please post the link to the MMS page for one of the hosts having the problem. (Don't worry: only you and employees of MongoDB Inc. can see the data located at that link.)
Why am I asking for this information? In my original reply, I posted a series of possible reasons. These diagnostics will either confirm or refute each possibility.
A) Missing indexes, or other schema design problems
- Checked by looking at the log file output
B) Working set exceeds available RAM
- Checked by looking at db.serverStatus() and 'top' output, also by looking at MMS
C) Inefficient pre-allocation due to using a filesystem other than ext4 or xfs
- Checked by looking at 'mount' output
D) Incorrect disk subsystem configuration
- Checked by looking at 'mdstat', 'blockdev' and 'lsblk' output (Other commands are used to orient me to your disk layout)
E) Other processes running on the host
- Checked by looking at 'top' output
F) Hard disk degradation or failure
- Checked by looking at 'iostat' output
Once I have this information, I can take the next step in the diagnosis. Based on your responses, I may have other questions to ask you. Have a great day!
-William
bruceguodong
Apr 16, 2014, 3:31:33 AM4/16/14
to mongod...@googlegroups.com
Hi, William:
Thanks for your patience and detailed diagnose instructions.
I will post some information about my db server and something others below.
1).
a. What distribution and version of Linux are you running?
$ uname -a
Linux dxyz169 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
b. Are you running on a physical or a virtual machine?
I am running on a physical machine.
c. Are you using physical disks or network disks?
I am using physical disks.
2). mongod.log
Because I set the variable diaglog's value to 0 and journal's to false, so I can not grep any useful log what you want.
The mostly log content is:
"Wed Apr 16 12:07:58.667 [conn10383528] authenticate db: reportdb { authenticate: 1, user: "***", nonce: "***", key: "***" }"
3).
$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 4.8T 309G 4.3T 7% /
udev 10M 0 10M 0% /dev
tmpfs 6.4G 216K 6.4G 1% /run
/dev/disk/by-uuid/78cf90ce-a5a9-4034-8ce2-17348ad0a56d 4.8T 309G 4.3T 7% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 39G 0 39G 0% /run/shm
$ mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8256580,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6606456k,mode=755)
/dev/disk/by-uuid/78cf90ce-a5a9-4034-8ce2-17348ad0a56d on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=40037180k)
$ du -sh $DATADIR
du -sh data/
292G data/
$ ls -l $DATADIR
drwxr-xr-x 3 db db 4096 Mar 24 12:04 admin
-rw-r--r-- 1 db db 0 Mar 24 11:53 diaglog.532fac27
-rw-r--r-- 1 db db 0 Mar 24 12:02 diaglog.532fae58
-rw-r--r-- 1 db db 0 Mar 24 12:03 diaglog.532fae89
-rw-r--r-- 1 db db 0 Mar 24 12:12 diaglog.532fb0b1
-rw-r--r-- 1 db db 0 Mar 24 12:14 diaglog.532fb119
drwxr-xr-x 2 db db 4096 Apr 16 10:56 journal
drwxr-xr-x 3 db db 4096 Mar 24 12:03 local
-rwxr-xr-x 1 db db 6 Mar 24 12:14 mongod.lock
drwxr-xr-x 3 db db 4096 Apr 14 07:49 reportdb
drwxr-xr-x 2 db db 4096 Apr 16 00:00 _tmp
$ sudo blockdev --report
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 5397163278336 /dev/sda
rw 256 512 4096 2048 1048576 /dev/sda1
rw 256 512 4096 4096 5259819745280 /dev/sda2
rw 256 512 4096 10273089536 137340387328 /dev/sda3
$ sudo cat /proc/mdstat
I can not get the kernel's RAID/md state.
Our disk raid level is RAID-6. I will attach the detail info snapshot.
$ sudo lsblk -a
sda 8:0 0 4.9T 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 4.8T 0 part /
└─sda3 8:3 0 127.9G 0 part [SWAP]
sr0 11:0 1 1024M 0 rom
loop0 7:0 0 0 loop
loop1 7:1 0 0 loop
loop2 7:2 0 0 loop
loop3 7:3 0 0 loop
loop4 7:4 0 0 loop
loop5 7:5 0 0 loop
loop6 7:6 0 0 loop
loop7 7:7 0 0 loop
Thanks for your patience and detailed diagnose instructions.
I will post some information about my db server and something others below.
1).
a. What distribution and version of Linux are you running?
$ uname -a
Linux dxyz169 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
b. Are you running on a physical or a virtual machine?
I am running on a physical machine.
c. Are you using physical disks or network disks?
I am using physical disks.
2). mongod.log
Because I set the variable diaglog's value to 0 and journal's to false, so I can not grep any useful log what you want.
The mostly log content is:
"Wed Apr 16 12:07:58.667 [conn10383528] authenticate db: reportdb { authenticate: 1, user: "***", nonce: "***", key: "***" }"
3).
$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 4.8T 309G 4.3T 7% /
udev 10M 0 10M 0% /dev
tmpfs 6.4G 216K 6.4G 1% /run
/dev/disk/by-uuid/78cf90ce-a5a9-4034-8ce2-17348ad0a56d 4.8T 309G 4.3T 7% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 39G 0 39G 0% /run/shm
$ mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8256580,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6606456k,mode=755)
/dev/disk/by-uuid/78cf90ce-a5a9-4034-8ce2-17348ad0a56d on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=40037180k)
$ du -sh $DATADIR
du -sh data/
292G data/
$ ls -l $DATADIR
drwxr-xr-x 3 db db 4096 Mar 24 12:04 admin
-rw-r--r-- 1 db db 0 Mar 24 11:53 diaglog.532fac27
-rw-r--r-- 1 db db 0 Mar 24 12:02 diaglog.532fae58
-rw-r--r-- 1 db db 0 Mar 24 12:03 diaglog.532fae89
-rw-r--r-- 1 db db 0 Mar 24 12:12 diaglog.532fb0b1
-rw-r--r-- 1 db db 0 Mar 24 12:14 diaglog.532fb119
drwxr-xr-x 2 db db 4096 Apr 16 10:56 journal
drwxr-xr-x 3 db db 4096 Mar 24 12:03 local
-rwxr-xr-x 1 db db 6 Mar 24 12:14 mongod.lock
drwxr-xr-x 3 db db 4096 Apr 14 07:49 reportdb
drwxr-xr-x 2 db db 4096 Apr 16 00:00 _tmp
$ sudo blockdev --report
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 5397163278336 /dev/sda
rw 256 512 4096 2048 1048576 /dev/sda1
rw 256 512 4096 4096 5259819745280 /dev/sda2
rw 256 512 4096 10273089536 137340387328 /dev/sda3
$ sudo cat /proc/mdstat
I can not get the kernel's RAID/md state.
Our disk raid level is RAID-6. I will attach the detail info snapshot.
$ sudo lsblk -a
sda 8:0 0 4.9T 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 4.8T 0 part /
└─sda3 8:3 0 127.9G 0 part [SWAP]
sr0 11:0 1 1024M 0 rom
loop0 7:0 0 0 loop
loop1 7:1 0 0 loop
loop2 7:2 0 0 loop
loop3 7:3 0 0 loop
loop4 7:4 0 0 loop
loop5 7:5 0 0 loop
loop6 7:6 0 0 loop
loop7 7:7 0 0 loop
$ mongo --eval 'printjson(db.serverStatus({workingSet:1}).workingSet)'
{
"note" : "thisIsAnEstimate",
"pagesInMemory" : 784714,
"computationTimeMicros" : 170390,
"overSeconds" : 404
}
$ mongo --eval 'printjson(db.serverStatus().mem)'
{
"bits" : 64,
"resident" : 65601,
"virtual" : 586774,
"supported" : true,
"mapped" : 292834,
"mappedWithJournal" : 585668
"note" : "thisIsAnEstimate",
"pagesInMemory" : 784714,
"computationTimeMicros" : 170390,
"overSeconds" : 404
}
$ mongo --eval 'printjson(db.serverStatus().mem)'
{
"bits" : 64,
"resident" : 65601,
"virtual" : 586774,
"supported" : true,
"mapped" : 292834,
"mappedWithJournal" : 585668
}
$ top -b -n 2
top - 14:24:32 up 25 days, 18:50, 3 users, load average: 0.30, 0.32, 0.29
Tasks: 184 total, 1 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.1 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 66064524 total, 65858960 used, 205564 free, 137544 buffers
KiB Swap: 134121468 total, 708 used, 134120760 free, 63016892 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14958 tempc 20 0 23168 1456 1060 R 6 0.0 0:00.01 top
1 root 20 0 10648 664 548 S 0 0.0 0:32.72 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:45.47 ksoftirqd/0
6 root rt 0 0 0 0 S 0 0.0 0:00.49 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:05.64 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:34.85 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:05.16 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:10.68 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:13.84 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:04.94 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:02.54 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:04.75 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:05.93 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:04.82 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:34.37 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:04.08 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:04.90 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:08.66 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:04.74 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/8
38 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:0
39 root 20 0 0 0 0 S 0 0.0 0:03.24 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:04.73 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/9
42 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:0
43 root 20 0 0 0 0 S 0 0.0 0:01.73 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:04.78 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.56 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:03.00 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:05.20 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:01.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:04.68 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.54 migration/13
58 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:0
59 root 20 0 0 0 0 S 0 0.0 0:00.13 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:06.10 migration/14
63 root 20 0 0 0 0 S 0 0.0 0:03.29 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:05.77 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:05.56 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.83 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:04.96 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:06.55 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:08.98 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:18.71 kworker/9:1
77 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/17
78 root rt 0 0 0 0 S 0 0.0 0:00.55 migration/18
79 root 20 0 0 0 0 S 0 0.0 0:14.51 kworker/18:0
80 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/18
81 root 20 0 0 0 0 S 0 0.0 0:22.35 kworker/8:1
82 root 20 0 0 0 0 S 0 0.0 0:23.55 kworker/7:1
83 root 20 0 0 0 0 S 0 0.0 0:24.35 kworker/6:1
84 root 20 0 0 0 0 S 0 0.0 0:39.76 kworker/5:1
85 root 20 0 0 0 0 S 0 0.0 0:32.81 kworker/4:1
86 root 20 0 0 0 0 S 0 0.0 0:18.46 kworker/3:1
87 root 20 0 0 0 0 S 0 0.0 0:28.42 kworker/2:1
89 root rt 0 0 0 0 S 0 0.0 0:04.44 watchdog/18
90 root rt 0 0 0 0 S 0 0.0 0:00.96 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:0
92 root 20 0 0 0 0 S 0 0.0 0:00.47 ksoftirqd/19
93 root rt 0 0 0 0 S 0 0.0 0:04.62 watchdog/19
94 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/20
95 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
96 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/20
97 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/20
98 root rt 0 0 0 0 S 0 0.0 0:00.47 migration/21
99 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
100 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/21
101 root rt 0 0 0 0 S 0 0.0 0:04.50 watchdog/21
102 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/22
103 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
104 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/22
105 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/22
106 root rt 0 0 0 0 S 0 0.0 0:00.50 migration/23
107 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
108 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/23
109 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/23
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
111 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
112 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
113 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
114 root 20 0 0 0 0 S 0 0.0 0:04.78 sync_supers
115 root 20 0 0 0 0 S 0 0.0 0:00.12 bdi-default
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
117 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
118 root 20 0 0 0 0 S 0 0.0 0:01.46 khungtaskd
119 root 20 0 0 0 0 S 0 0.0 0:59.04 kswapd0
120 root 20 0 0 0 0 S 0 0.0 0:43.05 kswapd1
121 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
122 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
123 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
124 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
128 root 20 0 0 0 0 S 0 0.0 1:37.76 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:15.87 kworker/21:1
130 root 20 0 0 0 0 S 0 0.0 0:18.05 kworker/19:1
131 root 20 0 0 0 0 S 0 0.0 0:18.61 kworker/17:1
132 root 20 0 0 0 0 S 0 0.0 0:20.74 kworker/10:1
133 root 20 0 0 0 0 S 0 0.0 0:14.23 kworker/20:1
134 root 20 0 0 0 0 S 0 0.0 0:12.89 kworker/23:1
135 root 20 0 0 0 0 S 0 0.0 0:11.63 kworker/22:1
136 root 20 0 0 0 0 S 0 0.0 0:18.76 kworker/11:1
138 root 20 0 0 0 0 S 0 0.0 1:45.11 kworker/14:1
139 root 20 0 0 0 0 S 0 0.0 0:13.14 kworker/12:1
140 root 20 0 0 0 0 S 0 0.0 0:18.48 kworker/16:1
321 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
323 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
345 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
410 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
411 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
412 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
413 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
414 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
415 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_6
419 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:4
420 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:5
459 root 20 0 0 0 0 S 0 0.0 4:07.42 jbd2/sda2-8
460 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
610 root 20 0 21592 1140 744 S 0 0.0 0:00.10 udevd
936 root 20 0 21588 684 440 S 0 0.0 0:00.00 udevd
937 root 20 0 21588 984 428 S 0 0.0 0:00.00 udevd
1238 root 20 0 0 0 0 S 0 0.0 0:14.17 kworker/13:2
2178 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:2
2459 root 20 0 116m 2468 988 S 0 0.0 0:02.38 rsyslogd
2534 messageb 20 0 29800 324 64 S 0 0.0 0:00.00 dbus-daemon
2552 ntp 20 0 38992 2200 1576 S 0 0.0 1:21.79 ntpd
2571 dnsmasq 20 0 30296 1768 584 S 0 0.0 0:13.25 dnsmasq
2572 root 20 0 0 0 0 S 0 0.0 15:05.44 flush-8:0
2593 root 20 0 4112 620 472 S 0 0.0 0:00.00 acpid
2652 root 20 0 49848 1164 556 S 0 0.0 0:00.02 sshd
2688 root 20 0 20404 1040 792 S 0 0.0 0:03.95 cron
2716 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2717 root 20 0 16252 780 620 S 0 0.0 0:00.00 getty
2718 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2719 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2720 root 20 0 16252 784 620 S 0 0.0 0:00.00 getty
2721 root 20 0 16252 772 620 S 0 0.0 0:00.00 getty
5380 root 20 0 71176 3552 2768 S 0 0.0 0:00.00 sshd
5403 ddblog 20 0 71176 1772 956 S 0 0.0 0:00.29 sshd
5405 ddblog 20 0 19460 2204 1652 S 0 0.0 0:00.01 bash
7652 nagios 20 0 25240 1192 672 S 0 0.0 0:51.83 nrpe
9134 ddblog 20 0 19468 2224 1664 S 0 0.0 0:00.00 bash
11039 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:2
11105 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:2
12113 root 20 0 37152 1408 1108 S 0 0.0 0:00.01 su
12116 tempc 20 0 19484 2256 1680 S 0 0.0 0:00.14 bash
14892 nagios 20 0 25892 2368 1224 S 0 0.0 0:00.00 nrpe
14893 nagios 20 0 25896 1144 0 S 0 0.0 0:00.00 nrpe
14894 nagios 20 0 10780 1452 1228 S 0 0.0 0:00.00 check_iostat
14920 nagios 20 0 10780 568 340 S 0 0.0 0:00.00 check_iostat
14921 nagios 20 0 5624 832 708 S 0 0.0 0:00.00 iostat
14922 nagios 20 0 6760 604 492 S 0 0.0 0:00.00 awk
17339 ddblog 20 0 19460 2012 1544 S 0 0.0 0:00.00 bash
17975 root 20 0 0 0 0 S 0 0.0 0:00.46 kworker/1:2
18072 ddblog 20 0 595m 16m 7940 S 0 0.0 0:00.20 mongo
19846 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
20403 root 20 0 0 0 0 S 0 0.0 0:13.47 kworker/0:1
22094 root 20 0 0 0 0 S 0 0.0 0:00.17 kworker/1:0
23197 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:2
25176 root 20 0 0 0 0 S 0 0.0 0:00.28 kworker/0:2
29063 tempc 20 0 573g 64g 63g S 0 101.7 3220:01 mongod
top - 14:24:35 up 25 days, 18:50, 3 users, load average: 0.36, 0.33, 0.29
Tasks: 178 total, 2 running, 176 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.2 us, 0.3 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 66064524 total, 65863172 used, 201352 free, 137544 buffers
KiB Swap: 134121468 total, 708 used, 134120760 free, 63017276 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29063 tempc 20 0 573g 64g 63g S 31 101.7 3220:02 mongod
138 root 20 0 0 0 0 R 0 0.0 1:45.12 kworker/14:1
5403 ddblog 20 0 71176 1772 956 S 0 0.0 0:00.30 sshd
14958 tempc 20 0 23176 1596 1156 R 0 0.0 0:00.02 top
1 root 20 0 10648 664 548 S 0 0.0 0:32.72 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:45.47 ksoftirqd/0
6 root rt 0 0 0 0 S 0 0.0 0:00.49 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:05.64 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:34.85 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:05.16 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:10.68 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:13.84 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:04.94 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:02.54 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:04.75 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:05.93 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:04.82 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:34.37 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:04.08 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:04.90 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:08.66 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:04.74 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/8
38 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:0
39 root 20 0 0 0 0 S 0 0.0 0:03.24 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:04.73 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/9
42 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:0
43 root 20 0 0 0 0 S 0 0.0 0:01.73 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:04.78 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.56 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:03.00 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:05.20 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:01.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:04.68 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.54 migration/13
58 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:0
59 root 20 0 0 0 0 S 0 0.0 0:00.13 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:06.10 migration/14
63 root 20 0 0 0 0 S 0 0.0 0:03.29 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:05.77 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:05.56 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.83 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:04.96 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:06.55 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:08.98 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:18.71 kworker/9:1
77 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/17
78 root rt 0 0 0 0 S 0 0.0 0:00.55 migration/18
79 root 20 0 0 0 0 S 0 0.0 0:14.51 kworker/18:0
80 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/18
81 root 20 0 0 0 0 S 0 0.0 0:22.35 kworker/8:1
82 root 20 0 0 0 0 S 0 0.0 0:23.55 kworker/7:1
83 root 20 0 0 0 0 S 0 0.0 0:24.35 kworker/6:1
84 root 20 0 0 0 0 S 0 0.0 0:39.76 kworker/5:1
85 root 20 0 0 0 0 S 0 0.0 0:32.81 kworker/4:1
86 root 20 0 0 0 0 S 0 0.0 0:18.46 kworker/3:1
87 root 20 0 0 0 0 S 0 0.0 0:28.42 kworker/2:1
89 root rt 0 0 0 0 S 0 0.0 0:04.44 watchdog/18
90 root rt 0 0 0 0 S 0 0.0 0:00.96 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:0
92 root 20 0 0 0 0 S 0 0.0 0:00.47 ksoftirqd/19
93 root rt 0 0 0 0 S 0 0.0 0:04.62 watchdog/19
94 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/20
95 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
96 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/20
97 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/20
98 root rt 0 0 0 0 S 0 0.0 0:00.47 migration/21
99 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
100 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/21
101 root rt 0 0 0 0 S 0 0.0 0:04.50 watchdog/21
102 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/22
103 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
104 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/22
105 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/22
106 root rt 0 0 0 0 S 0 0.0 0:00.50 migration/23
107 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
108 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/23
109 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/23
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
111 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
112 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
113 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
114 root 20 0 0 0 0 S 0 0.0 0:04.78 sync_supers
115 root 20 0 0 0 0 S 0 0.0 0:00.12 bdi-default
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
117 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
118 root 20 0 0 0 0 S 0 0.0 0:01.46 khungtaskd
119 root 20 0 0 0 0 S 0 0.0 0:59.04 kswapd0
120 root 20 0 0 0 0 S 0 0.0 0:43.05 kswapd1
121 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
122 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
123 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
124 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
128 root 20 0 0 0 0 S 0 0.0 1:37.76 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:15.87 kworker/21:1
130 root 20 0 0 0 0 S 0 0.0 0:18.05 kworker/19:1
131 root 20 0 0 0 0 S 0 0.0 0:18.61 kworker/17:1
132 root 20 0 0 0 0 S 0 0.0 0:20.74 kworker/10:1
133 root 20 0 0 0 0 S 0 0.0 0:14.23 kworker/20:1
134 root 20 0 0 0 0 S 0 0.0 0:12.89 kworker/23:1
135 root 20 0 0 0 0 S 0 0.0 0:11.63 kworker/22:1
136 root 20 0 0 0 0 S 0 0.0 0:18.76 kworker/11:1
139 root 20 0 0 0 0 S 0 0.0 0:13.14 kworker/12:1
140 root 20 0 0 0 0 S 0 0.0 0:18.48 kworker/16:1
321 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
323 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
345 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
410 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
411 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
412 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
413 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
414 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
415 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_6
419 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:4
420 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:5
459 root 20 0 0 0 0 S 0 0.0 4:07.42 jbd2/sda2-8
460 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
610 root 20 0 21592 1140 744 S 0 0.0 0:00.10 udevd
936 root 20 0 21588 684 440 S 0 0.0 0:00.00 udevd
937 root 20 0 21588 984 428 S 0 0.0 0:00.00 udevd
1238 root 20 0 0 0 0 S 0 0.0 0:14.17 kworker/13:2
2178 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:2
2459 root 20 0 116m 2468 988 S 0 0.0 0:02.38 rsyslogd
2534 messageb 20 0 29800 324 64 S 0 0.0 0:00.00 dbus-daemon
2552 ntp 20 0 38992 2200 1576 S 0 0.0 1:21.79 ntpd
2571 dnsmasq 20 0 30296 1768 584 S 0 0.0 0:13.25 dnsmasq
2572 root 20 0 0 0 0 S 0 0.0 15:05.44 flush-8:0
2593 root 20 0 4112 620 472 S 0 0.0 0:00.00 acpid
2652 root 20 0 49848 1164 556 S 0 0.0 0:00.02 sshd
2688 root 20 0 20404 1040 792 S 0 0.0 0:03.95 cron
2716 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2717 root 20 0 16252 780 620 S 0 0.0 0:00.00 getty
2718 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2719 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2720 root 20 0 16252 784 620 S 0 0.0 0:00.00 getty
2721 root 20 0 16252 772 620 S 0 0.0 0:00.00 getty
5380 root 20 0 71176 3552 2768 S 0 0.0 0:00.00 sshd
5405 ddblog 20 0 19460 2204 1652 S 0 0.0 0:00.01 bash
7652 nagios 20 0 25240 1192 672 S 0 0.0 0:51.83 nrpe
9134 ddblog 20 0 19468 2224 1664 S 0 0.0 0:00.00 bash
11039 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:2
11105 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:2
12113 root 20 0 37152 1408 1108 S 0 0.0 0:00.01 su
12116 tempc 20 0 19484 2256 1680 S 0 0.0 0:00.14 bash
17339 ddblog 20 0 19460 2012 1544 S 0 0.0 0:00.00 bash
17975 root 20 0 0 0 0 S 0 0.0 0:00.46 kworker/1:2
18072 ddblog 20 0 595m 16m 7940 S 0 0.0 0:00.20 mongo
19846 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
20403 root 20 0 0 0 0 S 0 0.0 0:13.47 kworker/0:1
22094 root 20 0 0 0 0 S 0 0.0 0:00.17 kworker/1:0
23197 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:2
25176 root 20 0 0 0 0 S 0 0.0 0:00.28 kworker/0:2
Tasks: 184 total, 1 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.1 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 66064524 total, 65858960 used, 205564 free, 137544 buffers
KiB Swap: 134121468 total, 708 used, 134120760 free, 63016892 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14958 tempc 20 0 23168 1456 1060 R 6 0.0 0:00.01 top
1 root 20 0 10648 664 548 S 0 0.0 0:32.72 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:45.47 ksoftirqd/0
6 root rt 0 0 0 0 S 0 0.0 0:00.49 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:05.64 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:34.85 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:05.16 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:10.68 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:13.84 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:04.94 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:02.54 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:04.75 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:05.93 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:04.82 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:34.37 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:04.08 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:04.90 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:08.66 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:04.74 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/8
38 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:0
39 root 20 0 0 0 0 S 0 0.0 0:03.24 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:04.73 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/9
42 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:0
43 root 20 0 0 0 0 S 0 0.0 0:01.73 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:04.78 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.56 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:03.00 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:05.20 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:01.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:04.68 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.54 migration/13
58 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:0
59 root 20 0 0 0 0 S 0 0.0 0:00.13 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:06.10 migration/14
63 root 20 0 0 0 0 S 0 0.0 0:03.29 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:05.77 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:05.56 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.83 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:04.96 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:06.55 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:08.98 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:18.71 kworker/9:1
77 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/17
78 root rt 0 0 0 0 S 0 0.0 0:00.55 migration/18
79 root 20 0 0 0 0 S 0 0.0 0:14.51 kworker/18:0
80 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/18
81 root 20 0 0 0 0 S 0 0.0 0:22.35 kworker/8:1
82 root 20 0 0 0 0 S 0 0.0 0:23.55 kworker/7:1
83 root 20 0 0 0 0 S 0 0.0 0:24.35 kworker/6:1
84 root 20 0 0 0 0 S 0 0.0 0:39.76 kworker/5:1
85 root 20 0 0 0 0 S 0 0.0 0:32.81 kworker/4:1
86 root 20 0 0 0 0 S 0 0.0 0:18.46 kworker/3:1
87 root 20 0 0 0 0 S 0 0.0 0:28.42 kworker/2:1
89 root rt 0 0 0 0 S 0 0.0 0:04.44 watchdog/18
90 root rt 0 0 0 0 S 0 0.0 0:00.96 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:0
92 root 20 0 0 0 0 S 0 0.0 0:00.47 ksoftirqd/19
93 root rt 0 0 0 0 S 0 0.0 0:04.62 watchdog/19
94 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/20
95 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
96 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/20
97 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/20
98 root rt 0 0 0 0 S 0 0.0 0:00.47 migration/21
99 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
100 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/21
101 root rt 0 0 0 0 S 0 0.0 0:04.50 watchdog/21
102 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/22
103 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
104 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/22
105 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/22
106 root rt 0 0 0 0 S 0 0.0 0:00.50 migration/23
107 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
108 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/23
109 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/23
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
111 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
112 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
113 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
114 root 20 0 0 0 0 S 0 0.0 0:04.78 sync_supers
115 root 20 0 0 0 0 S 0 0.0 0:00.12 bdi-default
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
117 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
118 root 20 0 0 0 0 S 0 0.0 0:01.46 khungtaskd
119 root 20 0 0 0 0 S 0 0.0 0:59.04 kswapd0
120 root 20 0 0 0 0 S 0 0.0 0:43.05 kswapd1
121 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
122 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
123 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
124 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
128 root 20 0 0 0 0 S 0 0.0 1:37.76 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:15.87 kworker/21:1
130 root 20 0 0 0 0 S 0 0.0 0:18.05 kworker/19:1
131 root 20 0 0 0 0 S 0 0.0 0:18.61 kworker/17:1
132 root 20 0 0 0 0 S 0 0.0 0:20.74 kworker/10:1
133 root 20 0 0 0 0 S 0 0.0 0:14.23 kworker/20:1
134 root 20 0 0 0 0 S 0 0.0 0:12.89 kworker/23:1
135 root 20 0 0 0 0 S 0 0.0 0:11.63 kworker/22:1
136 root 20 0 0 0 0 S 0 0.0 0:18.76 kworker/11:1
138 root 20 0 0 0 0 S 0 0.0 1:45.11 kworker/14:1
139 root 20 0 0 0 0 S 0 0.0 0:13.14 kworker/12:1
140 root 20 0 0 0 0 S 0 0.0 0:18.48 kworker/16:1
321 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
323 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
345 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
410 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
411 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
412 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
413 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
414 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
415 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_6
419 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:4
420 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:5
459 root 20 0 0 0 0 S 0 0.0 4:07.42 jbd2/sda2-8
460 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
610 root 20 0 21592 1140 744 S 0 0.0 0:00.10 udevd
936 root 20 0 21588 684 440 S 0 0.0 0:00.00 udevd
937 root 20 0 21588 984 428 S 0 0.0 0:00.00 udevd
1238 root 20 0 0 0 0 S 0 0.0 0:14.17 kworker/13:2
2178 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:2
2459 root 20 0 116m 2468 988 S 0 0.0 0:02.38 rsyslogd
2534 messageb 20 0 29800 324 64 S 0 0.0 0:00.00 dbus-daemon
2552 ntp 20 0 38992 2200 1576 S 0 0.0 1:21.79 ntpd
2571 dnsmasq 20 0 30296 1768 584 S 0 0.0 0:13.25 dnsmasq
2572 root 20 0 0 0 0 S 0 0.0 15:05.44 flush-8:0
2593 root 20 0 4112 620 472 S 0 0.0 0:00.00 acpid
2652 root 20 0 49848 1164 556 S 0 0.0 0:00.02 sshd
2688 root 20 0 20404 1040 792 S 0 0.0 0:03.95 cron
2716 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2717 root 20 0 16252 780 620 S 0 0.0 0:00.00 getty
2718 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2719 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2720 root 20 0 16252 784 620 S 0 0.0 0:00.00 getty
2721 root 20 0 16252 772 620 S 0 0.0 0:00.00 getty
5380 root 20 0 71176 3552 2768 S 0 0.0 0:00.00 sshd
5403 ddblog 20 0 71176 1772 956 S 0 0.0 0:00.29 sshd
5405 ddblog 20 0 19460 2204 1652 S 0 0.0 0:00.01 bash
7652 nagios 20 0 25240 1192 672 S 0 0.0 0:51.83 nrpe
9134 ddblog 20 0 19468 2224 1664 S 0 0.0 0:00.00 bash
11039 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:2
11105 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:2
12113 root 20 0 37152 1408 1108 S 0 0.0 0:00.01 su
12116 tempc 20 0 19484 2256 1680 S 0 0.0 0:00.14 bash
14892 nagios 20 0 25892 2368 1224 S 0 0.0 0:00.00 nrpe
14893 nagios 20 0 25896 1144 0 S 0 0.0 0:00.00 nrpe
14894 nagios 20 0 10780 1452 1228 S 0 0.0 0:00.00 check_iostat
14920 nagios 20 0 10780 568 340 S 0 0.0 0:00.00 check_iostat
14921 nagios 20 0 5624 832 708 S 0 0.0 0:00.00 iostat
14922 nagios 20 0 6760 604 492 S 0 0.0 0:00.00 awk
17339 ddblog 20 0 19460 2012 1544 S 0 0.0 0:00.00 bash
17975 root 20 0 0 0 0 S 0 0.0 0:00.46 kworker/1:2
18072 ddblog 20 0 595m 16m 7940 S 0 0.0 0:00.20 mongo
19846 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
20403 root 20 0 0 0 0 S 0 0.0 0:13.47 kworker/0:1
22094 root 20 0 0 0 0 S 0 0.0 0:00.17 kworker/1:0
23197 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:2
25176 root 20 0 0 0 0 S 0 0.0 0:00.28 kworker/0:2
29063 tempc 20 0 573g 64g 63g S 0 101.7 3220:01 mongod
top - 14:24:35 up 25 days, 18:50, 3 users, load average: 0.36, 0.33, 0.29
Tasks: 178 total, 2 running, 176 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.2 us, 0.3 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 66064524 total, 65863172 used, 201352 free, 137544 buffers
KiB Swap: 134121468 total, 708 used, 134120760 free, 63017276 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29063 tempc 20 0 573g 64g 63g S 31 101.7 3220:02 mongod
138 root 20 0 0 0 0 R 0 0.0 1:45.12 kworker/14:1
5403 ddblog 20 0 71176 1772 956 S 0 0.0 0:00.30 sshd
14958 tempc 20 0 23176 1596 1156 R 0 0.0 0:00.02 top
1 root 20 0 10648 664 548 S 0 0.0 0:32.72 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:45.47 ksoftirqd/0
6 root rt 0 0 0 0 S 0 0.0 0:00.49 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:05.64 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/1
10 root 20 0 0 0 0 S 0 0.0 0:34.85 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:05.16 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:10.68 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:13.84 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:04.94 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:02.54 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:04.75 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:05.93 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:04.82 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:34.37 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:04.08 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:04.90 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/7
35 root 20 0 0 0 0 S 0 0.0 0:08.66 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:04.74 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/8
38 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:0
39 root 20 0 0 0 0 S 0 0.0 0:03.24 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:04.73 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/9
42 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:0
43 root 20 0 0 0 0 S 0 0.0 0:01.73 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:04.78 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.56 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:03.00 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:05.20 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.40 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:01.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:04.68 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.54 migration/13
58 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:0
59 root 20 0 0 0 0 S 0 0.0 0:00.13 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:04.55 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:06.10 migration/14
63 root 20 0 0 0 0 S 0 0.0 0:03.29 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:05.77 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:05.56 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.83 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:04.96 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:06.55 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:08.98 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:18.71 kworker/9:1
77 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/17
78 root rt 0 0 0 0 S 0 0.0 0:00.55 migration/18
79 root 20 0 0 0 0 S 0 0.0 0:14.51 kworker/18:0
80 root 20 0 0 0 0 S 0 0.0 0:00.32 ksoftirqd/18
81 root 20 0 0 0 0 S 0 0.0 0:22.35 kworker/8:1
82 root 20 0 0 0 0 S 0 0.0 0:23.55 kworker/7:1
83 root 20 0 0 0 0 S 0 0.0 0:24.35 kworker/6:1
84 root 20 0 0 0 0 S 0 0.0 0:39.76 kworker/5:1
85 root 20 0 0 0 0 S 0 0.0 0:32.81 kworker/4:1
86 root 20 0 0 0 0 S 0 0.0 0:18.46 kworker/3:1
87 root 20 0 0 0 0 S 0 0.0 0:28.42 kworker/2:1
89 root rt 0 0 0 0 S 0 0.0 0:04.44 watchdog/18
90 root rt 0 0 0 0 S 0 0.0 0:00.96 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:0
92 root 20 0 0 0 0 S 0 0.0 0:00.47 ksoftirqd/19
93 root rt 0 0 0 0 S 0 0.0 0:04.62 watchdog/19
94 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/20
95 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
96 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/20
97 root rt 0 0 0 0 S 0 0.0 0:05.02 watchdog/20
98 root rt 0 0 0 0 S 0 0.0 0:00.47 migration/21
99 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
100 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/21
101 root rt 0 0 0 0 S 0 0.0 0:04.50 watchdog/21
102 root rt 0 0 0 0 S 0 0.0 0:00.42 migration/22
103 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
104 root 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/22
105 root rt 0 0 0 0 S 0 0.0 0:04.52 watchdog/22
106 root rt 0 0 0 0 S 0 0.0 0:00.50 migration/23
107 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
108 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/23
109 root rt 0 0 0 0 S 0 0.0 0:04.48 watchdog/23
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
111 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
112 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
113 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
114 root 20 0 0 0 0 S 0 0.0 0:04.78 sync_supers
115 root 20 0 0 0 0 S 0 0.0 0:00.12 bdi-default
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
117 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
118 root 20 0 0 0 0 S 0 0.0 0:01.46 khungtaskd
119 root 20 0 0 0 0 S 0 0.0 0:59.04 kswapd0
120 root 20 0 0 0 0 S 0 0.0 0:43.05 kswapd1
121 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
122 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
123 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
124 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
128 root 20 0 0 0 0 S 0 0.0 1:37.76 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:15.87 kworker/21:1
130 root 20 0 0 0 0 S 0 0.0 0:18.05 kworker/19:1
131 root 20 0 0 0 0 S 0 0.0 0:18.61 kworker/17:1
132 root 20 0 0 0 0 S 0 0.0 0:20.74 kworker/10:1
133 root 20 0 0 0 0 S 0 0.0 0:14.23 kworker/20:1
134 root 20 0 0 0 0 S 0 0.0 0:12.89 kworker/23:1
135 root 20 0 0 0 0 S 0 0.0 0:11.63 kworker/22:1
136 root 20 0 0 0 0 S 0 0.0 0:18.76 kworker/11:1
139 root 20 0 0 0 0 S 0 0.0 0:13.14 kworker/12:1
140 root 20 0 0 0 0 S 0 0.0 0:18.48 kworker/16:1
321 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
323 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
345 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
410 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
411 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
412 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3
413 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4
414 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_5
415 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_6
419 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:4
420 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:5
459 root 20 0 0 0 0 S 0 0.0 4:07.42 jbd2/sda2-8
460 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
610 root 20 0 21592 1140 744 S 0 0.0 0:00.10 udevd
936 root 20 0 21588 684 440 S 0 0.0 0:00.00 udevd
937 root 20 0 21588 984 428 S 0 0.0 0:00.00 udevd
1238 root 20 0 0 0 0 S 0 0.0 0:14.17 kworker/13:2
2178 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:2
2459 root 20 0 116m 2468 988 S 0 0.0 0:02.38 rsyslogd
2534 messageb 20 0 29800 324 64 S 0 0.0 0:00.00 dbus-daemon
2552 ntp 20 0 38992 2200 1576 S 0 0.0 1:21.79 ntpd
2571 dnsmasq 20 0 30296 1768 584 S 0 0.0 0:13.25 dnsmasq
2572 root 20 0 0 0 0 S 0 0.0 15:05.44 flush-8:0
2593 root 20 0 4112 620 472 S 0 0.0 0:00.00 acpid
2652 root 20 0 49848 1164 556 S 0 0.0 0:00.02 sshd
2688 root 20 0 20404 1040 792 S 0 0.0 0:03.95 cron
2716 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2717 root 20 0 16252 780 620 S 0 0.0 0:00.00 getty
2718 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2719 root 20 0 16252 776 620 S 0 0.0 0:00.00 getty
2720 root 20 0 16252 784 620 S 0 0.0 0:00.00 getty
2721 root 20 0 16252 772 620 S 0 0.0 0:00.00 getty
5380 root 20 0 71176 3552 2768 S 0 0.0 0:00.00 sshd
5405 ddblog 20 0 19460 2204 1652 S 0 0.0 0:00.01 bash
7652 nagios 20 0 25240 1192 672 S 0 0.0 0:51.83 nrpe
9134 ddblog 20 0 19468 2224 1664 S 0 0.0 0:00.00 bash
11039 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:2
11105 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:2
12113 root 20 0 37152 1408 1108 S 0 0.0 0:00.01 su
12116 tempc 20 0 19484 2256 1680 S 0 0.0 0:00.14 bash
17339 ddblog 20 0 19460 2012 1544 S 0 0.0 0:00.00 bash
17975 root 20 0 0 0 0 S 0 0.0 0:00.46 kworker/1:2
18072 ddblog 20 0 595m 16m 7940 S 0 0.0 0:00.20 mongo
19846 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
20403 root 20 0 0 0 0 S 0 0.0 0:13.47 kworker/0:1
22094 root 20 0 0 0 0 S 0 0.0 0:00.17 kworker/1:0
23197 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/6:2
25176 root 20 0 0 0 0 S 0 0.0 0:00.28 kworker/0:2
$ iostat -xtm 1 3
(Here I show the 10 counts)
iostat -xtm 1 10
Linux 3.2.0-4-amd64 (dxyz169) Wednesday, April 16, 2014 _x86_64_ (24 CPU)
Wednesday, April 16, 2014 02:33:52 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.32 0.00 0.09 0.02 0.00 99.56
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 5.82 0.22 46.82 0.00 0.42 18.33 0.67 14.33 4.82 14.37 0.14 0.66
Wednesday, April 16, 2014 02:33:53 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.33 0.92 0.00 98.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 2990.00 0.00 19.12 13.10 39.21 12.49 0.00 12.49 0.10 30.40
Wednesday, April 16, 2014 02:33:54 CST
avg-cpu: %user %nice %system %iowait %steal %idle
2.51 0.00 0.58 0.00 0.00 96.91
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 113.00 0.00 1.18 21.31 0.70 22.87 0.00 22.87 1.45 16.40
Wednesday, April 16, 2014 02:33:55 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 0.17 0.00 0.00 99.21
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 8.00 0.00 0.12 32.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:56 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.04 0.00 0.00 99.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 2.00 0.00 0.01 8.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:57 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.04 0.00 0.00 0.00 0.00 99.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:58 CST
avg-cpu: %user %nice %system %iowait %steal %idle
2.51 0.00 0.54 0.00 0.00 96.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 7.00 0.00 23.00 0.00 0.62 55.65 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:59 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 0.29 0.00 0.00 99.08
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 10.00 0.00 0.14 28.80 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:34:00 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:34:01 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.00 0.00 99.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 8.00 0.00 0.13 34.00 0.00 0.00 0.00 0.00 0.00 0.00
4).
indexes' problem because of our warning value is set to too low, now is ok.
last_flush_time item still be a problem.
disply the nagios's output:
last_flush_time 1.0927 kms Last 2.7716 kms Max 0.8125 kms Average
Warning: 1100.0
Critical: 1500.0
5).
Our systemes are not in MMS. Sorry about that there is not any useful infomation about what you want.
Hope the infomation above can be useful, and thanks for your help very much.
在 2014年4月14日星期一UTC+8下午5时41分02秒，bruceguodong写道：
iostat -xtm 1 10
Linux 3.2.0-4-amd64 (dxyz169) Wednesday, April 16, 2014 _x86_64_ (24 CPU)
Wednesday, April 16, 2014 02:33:52 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.32 0.00 0.09 0.02 0.00 99.56
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 5.82 0.22 46.82 0.00 0.42 18.33 0.67 14.33 4.82 14.37 0.14 0.66
Wednesday, April 16, 2014 02:33:53 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.33 0.92 0.00 98.75
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 2990.00 0.00 19.12 13.10 39.21 12.49 0.00 12.49 0.10 30.40
Wednesday, April 16, 2014 02:33:54 CST
avg-cpu: %user %nice %system %iowait %steal %idle
2.51 0.00 0.58 0.00 0.00 96.91
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 113.00 0.00 1.18 21.31 0.70 22.87 0.00 22.87 1.45 16.40
Wednesday, April 16, 2014 02:33:55 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 0.17 0.00 0.00 99.21
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 8.00 0.00 0.12 32.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:56 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.04 0.00 0.00 99.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 2.00 0.00 0.01 8.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:57 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.04 0.00 0.00 0.00 0.00 99.96
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:58 CST
avg-cpu: %user %nice %system %iowait %steal %idle
2.51 0.00 0.54 0.00 0.00 96.95
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 7.00 0.00 23.00 0.00 0.62 55.65 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:33:59 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 0.29 0.00 0.00 99.08
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 10.00 0.00 0.14 28.80 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:34:00 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Wednesday, April 16, 2014 02:34:01 CST
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.00 0.00 99.58
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 8.00 0.00 0.13 34.00 0.00 0.00 0.00 0.00 0.00 0.00
4).
indexes' problem because of our warning value is set to too low, now is ok.
last_flush_time item still be a problem.
disply the nagios's output:
last_flush_time 1.0927 kms Last 2.7716 kms Max 0.8125 kms Average
Warning: 1100.0
Critical: 1500.0
5).
Our systemes are not in MMS. Sorry about that there is not any useful infomation about what you want.
Hope the infomation above can be useful, and thanks for your help very much.
在 2014年4月14日星期一UTC+8下午5时41分02秒，bruceguodong写道：
William Zola
Apr 17, 2014, 8:43:22 AM4/17/14
to mongod...@googlegroups.com
Hi Bruce!
Thanks for the data! Based on it, I have just a few more diagnostic commands that I'd like you to run. Please post the output of the following commands:
$ mongo --eval 'printjson(db.adminCommand("getCmdLineOpts"))'
$ mongo --eval 'printjson(db.adminCommand({getParameter:"*"}))'
$ mongo --eval 'printjson(db.serverStatus().backgroundFlushing)'$ mongo --eval 'printjson(db.serverStatus().dur)'Please run the last two commands during a time when you're seeing the high disk I/O
At this point, I'm reasonably sure that once I have this information I'll be able to explain what you're seeing.
Have a great day!
-William
[snip]
$ iostat -xtm 1 3
(Here I show the 10 counts)
iostat -xtm 1 10
Linux 3.2.0-4-amd64 (dxyz169) Wednesday, April 16, 2014 _x86_64_ (24 CPU)
[snip]
bruceguodong
Apr 18, 2014, 3:21:49 AM4/18/14
to mongod...@googlegroups.com
Hi, William:
Thans for your time.
Here are the output of the command that you gave to me.
$ mongo --eval 'printjson(db.adminCommand("getCmdLineOpts"))'
{
"argv" : [
"mongod",
"-f",
"**/mongodb/conf/mongodb.conf"
],
"parsed" : {
"auth" : "true",
"bind_ip" : "**",
"config" : "**/mongodb/conf/mongodb.conf",
"dbpath" : "**/mongodb/data",
"diaglog" : 0,
"directoryperdb" : "true",
"fork" : "true",
"keyFile" : "**/mongodb/key/keyfile",
"logappend" : "true",
"logpath" : "**/mongodb/logs/mongod.log",
"nounixsocket" : "true",
"port" : 27017,
"quiet" : "true",
"replSet" : "Set0",
"rest" : "true"
},
"ok" : 1
}
$ mongo --eval 'printjson(db.adminCommand({getParameter:"*"}))'
{
"authOnPrimaryOnly" : true,
"enableLocalhostAuthBypass" : true,
"enableTestCommands" : 0,
"ignoreInitialVersionFailure" : false,
"logLevel" : 0,
"logUserIds" : false,
"notablescan" : false,
"quiet" : true,
"releaseConnectionsAfterResponse" : false,
"replApplyBatchSize" : 1,
"replIndexPrefetch" : "all",
"supportCompatibilityFormPrivilegeDocuments" : true,
"syncdelay" : 60,
"textSearchEnabled" : false,
"ttlMonitorEnabled" : true,
"ok" : 1
}
$ mongo --eval 'printjson(db.serverStatus().backgroundFlushing)'
$ mongo --eval 'printjson(db.serverStatus().dur)'
Set0:PRIMARY> db.serverStatus().backgroundFlushing
{
"flushes" : 35985,
"total_ms" : 9032957,
"average_ms" : 251.02006391552035,
"last_ms" : 490,
"last_finished" : ISODate("2014-04-18T03:59:20.193Z")
}
Set0:PRIMARY> db.serverStatus().dur
{
"commits" : 26,
"journaledMB" : 0.720896,
"writeToDataFilesMB" : 1.373403,
"compression" : 0.4843591897067222,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3093,
"prepLogBuffer" : 4,
"writeToJournal" : 369,
"writeToDataFiles" : 57,
"remapPrivateView" : 2
}
}
Set0:PRIMARY>
Set0:PRIMARY> db.serverStatus().backgroundFlushing
{
"flushes" : 36043,
"total_ms" : 9077333,
"average_ms" : 251.84732125516743,
"last_ms" : 1417,
"last_finished" : ISODate("2014-04-18T04:57:21.126Z")
}
Set0:PRIMARY> db.serverStatus().dur
{
"commits" : 30,
"journaledMB" : 0.8192,
"writeToDataFilesMB" : 1.519397,
"compression" : 0.5022198953136361,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3099,
"prepLogBuffer" : 4,
"writeToJournal" : 17,
"writeToDataFiles" : 8,
"remapPrivateView" : 1
}
}
Set0:PRIMARY> db.serverStatus().backgroundFlushing
{
"flushes" : 36057,
"total_ms" : 9090970,
"average_ms" : 252.1277421859833,
"last_ms" : 1011,
"last_finished" : ISODate("2014-04-18T05:11:20.720Z")
}
Set0:PRIMARY> db.serverStatus().dur
{
"commits" : 30,
"journaledMB" : 0.090112,
"writeToDataFilesMB" : 0.131226,
"compression" : 0.6205675956724446,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3074,
"prepLogBuffer" : 0,
"writeToJournal" : 2,
"writeToDataFiles" : 1,
"remapPrivateView" : 3
}
}
# Secondary:
Set0:SECONDARY> db.serverStatus().backgroundFlushing
{
"flushes" : 36057,
"total_ms" : 7671313,
"average_ms" : 212.75516543250964,
"last_ms" : 349,
"last_finished" : ISODate("2014-04-18T05:12:00.558Z")
}
Set0:SECONDARY> db.serverStatus().dur
{
"commits" : 30,
"journaledMB" : 0.581632,
"writeToDataFilesMB" : 1.123692,
"compression" : 0.47226834668890383,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3094,
"prepLogBuffer" : 4,
"writeToJournal" : 13,
"writeToDataFiles" : 7,
"remapPrivateView" : 1
}
}
Set0:SECONDARY> db.serverStatus().backgroundFlushing
{
"flushes" : 36160,
"total_ms" : 7735806,
"average_ms" : 213.93268805309734,
"last_ms" : 534,
"last_finished" : ISODate("2014-04-18T06:55:00.754Z")
}
Set0:SECONDARY> db.serverStatus().dur
{
"commits" : 30,
"journaledMB" : 0.475136,
"writeToDataFilesMB" : 0.837285,
"compression" : 0.5229584592660426,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3088,
"prepLogBuffer" : 2,
"writeToJournal" : 10,
"writeToDataFiles" : 5,
"remapPrivateView" : 2
}
}
I show you a little more server status' data, hope that can help make your conclusion.
在 2014年4月17日星期四UTC+8下午8时43分22秒，William Zola写道：
William Zola
Apr 30, 2014, 12:03:11 PM4/30/14
to mongod...@googlegroups.com
Hi Bruce!
Thanks for the extra information. The good news is that your system is not under distress, and the spikes in disk I/O that you're seeing are part of MongoDB normal operation.
Looking at the iostat output, we see that your disks are mostly idle (0 writes per second and 0 reads per second), but that there is a single spike at 02:33:53 CST where your system writes roughly 20 MB of data to disk, and then returns to being idle in two seconds.
The most likely cause of this behavior is that you're seeing the effect of the background flush. MongoDB will, by default, flush all in-RAM changes to disk once every 60 seconds. By looking at the output of the 'getParameter' command, I can see that you have not changed this default. ( "syncdelay" : 60 )
So what appears to be happening is that MongoDB is flushing the in-memory changes to the data files, and that is what is causing the I/O spike. This spike is quite minor, and there's lots of evidence to back this up.
1) MongoDB has not had to update the working set via a page-in for over 400 seconds (that's right around 6.5 minutes!) We can see this from db.serverStatus({workingSet:1}).workingSet.overSeconds This means that your working set fits entirely in RAM, so you're not experiencing memory pressure.
2) The average background flush is taking roughly 1/4 of a second. We can see this from db.serverStatus().backgroundFlushing.average_ms, which is just under 252 ms.
3) The highest reported time for background flush from the examples you sent me is right around 1.5 seconds. We can see this from db.serverStatus().backgroundFlushing.last_ms, which is 1417 ms in the highest example you posted (just under 1.5 seconds)
I think your nagios settings might be too aggressive. You posted this output from nagios:
last_flush_time 1.0927 kms Last 2.7716 kms Max 0.8125 kms Average
Warning: 1100.0
Critical: 1500.0
According to this, your maximum background flush time was just under 3 seconds, and the average is 0.8 seconds, which is in line with what the serverStatus command was showing.
Also, this seems to show that nagios is configured to warn when the last flush takes more than 1.1 seconds. This is *way* to aggressive. I'd suggest setting the warning to be at 15 seconds, and critical to be at 30.
I hope you found this useful!
-William
References:
Bruce Feng
May 6, 2014, 2:32:28 AM5/6/14
to mongod...@googlegroups.com
Hi, William:
I apologize for not saying thanks to you for your thorough explaination immediately.
I just come back from home for some personal reasons.
Follow your suggestion, I will change the settings of the monitor items to a higher
value and watch for the server status real-time.
Next I will read more documents about MongoDB for deeper understanding.
And last, thanks for your help again, William.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/TSvzirKxd94/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/966f7463-44fd-4b68-a5d1-a50cb58fde51%40googlegroups.com.You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/TSvzirKxd94/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
s.molinari
May 6, 2014, 10:53:56 AM5/6/14
to mongod...@googlegroups.com
Where is the Google+ button when you need it?:-)
Scott
Reply all
Reply to author
Forward
0 new messages