Mongodb 2.6.7 memory issue: Out of Memory error, mongod invoked oom-killer

[Ben]

We recently upgraded our sharded mongo cluster from version 2.4 to 2.6.7, and now we are experiencing memory errors that cause the mongod process to die. Prior to the upgrade we never experienced any memory issues, especially one that killed the mongod process. The amount of data in our dbs and the available RAM (7.5 GiB) also did not change during the upgrade. We are running Ubuntu 12.04 machines. Our first efforts were to set the maxConnection back to 20000 because in v 2.4 connections were caped at 20k, but 2.6 took that cap away. We also added a 5 GiB swap partition to help with memory. These efforts helped reduce the frequency of our memory issues, but did not complete stop them. Are there an known memory leaks in 2.6.7 that could be the cause?  I'm providing all relevant info in our syslog and mongod.log at the time of the last Out of Memory error. 

syslog:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840832] mongod invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840837] mongod cpuset=/ mems_allowed=0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840840] Pid: 20566, comm: mongod Not tainted 3.2.0-29-virtual #46-Ubuntu

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840842] Call Trace:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840850]  [<ffffffff810bd5fd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840854]  [<ffffffff81117ce1>] dump_header+0x91/0xe0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840856]  [<ffffffff81118065>] oom_kill_process+0x85/0xb0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840858]  [<ffffffff8111840a>] out_of_memory+0xfa/0x220

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840861]  [<ffffffff8111de3a>] __alloc_pages_nodemask+0x7ea/0x800

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840865]  [<ffffffff8165296e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840869]  [<ffffffff81154b73>] alloc_pages_current+0xa3/0x110

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840873]  [<ffffffff81114a4f>] __page_cache_alloc+0x8f/0xa0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840875]  [<ffffffff81114d4e>] ? find_get_page+0x1e/0x90

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840878]  [<ffffffff81116bd2>] filemap_fault+0x212/0x3c0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840881]  [<ffffffff81136e62>] __do_fault+0x72/0x550

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840883]  [<ffffffff8113a70a>] handle_pte_fault+0xfa/0x200

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840887]  [<ffffffff8100648e>] ? xen_pmd_val+0xe/0x10

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840889]  [<ffffffff810052e9>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840891]  [<ffffffff8113abc8>] handle_mm_fault+0x1f8/0x350

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840894]  [<ffffffff8165625b>] do_page_fault+0x14b/0x520

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840897]  [<ffffffff8103cda5>] ? pvclock_clocksource_read+0x55/0xf0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840900]  [<ffffffff8100a640>] ? xen_clocksource_read+0x20/0x30

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840902]  [<ffffffff8100a729>] ? xen_clocksource_get_cycles+0x9/0x10

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840906]  [<ffffffff81093177>] ? getnstimeofday+0x57/0xe0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840908]  [<ffffffff81652eb5>] page_fault+0x25/0x30

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840910] Mem-Info:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840911] Node 0 DMA per-cpu:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840913] CPU    0: hi:    0, btch:   1 usd:   0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840915] CPU    1: hi:    0, btch:   1 usd:   0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840916] Node 0 DMA32 per-cpu:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840917] CPU    0: hi:  186, btch:  31 usd:  11

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840919] CPU    1: hi:  186, btch:  31 usd:  30

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840920] Node 0 Normal per-cpu:

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840921] CPU    0: hi:  186, btch:  31 usd: 176

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840923] CPU    1: hi:  186, btch:  31 usd:  59

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840926] active_anon:1373142 inactive_anon:274707 isolated_anon:0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840926]  active_file:357 inactive_file:560 isolated_file:0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840927]  unevictable:8 dirty:0 writeback:0 unstable:0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840928]  free:8273 slab_reclaimable:5553 slab_unreclaimable:4715

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840928]  mapped:243 shmem:8 pagetables:211684 bounce:0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840930] Node 0 DMA free:7856kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7600kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840935] lowmem_reserve[]: 0 4016 7552 7552

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840938] Node 0 DMA32 free:20044kB min:5912kB low:7388kB high:8868kB active_anon:2993784kB inactive_anon:599060kB active_file:400kB inactive_file:1120kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:32kB dirty:0kB writeback:0kB mapped:372kB shmem:8kB slab_reclaimable:10148kB slab_unreclaimable:7480kB kernel_stack:6024kB pagetables:436136kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1526 all_unreclaimable? yes

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840945] lowmem_reserve[]: 0 0 3535 3535

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840948] Node 0 Normal free:5192kB min:5204kB low:6504kB high:7804kB active_anon:2498784kB inactive_anon:499768kB active_file:1028kB inactive_file:1120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3620736kB mlocked:0kB dirty:0kB writeback:0kB mapped:600kB shmem:24kB slab_reclaimable:12064kB slab_unreclaimable:11380kB kernel_stack:1864kB pagetables:410600kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3860 all_unreclaimable? yes

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840954] lowmem_reserve[]: 0 0 0 0

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840956] Node 0 DMA: 2*4kB 3*8kB 1*16kB 2*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 0*4096kB = 7856kB

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840964] Node 0 DMA32: 3879*4kB 50*8kB 134*16kB 2*32kB 0*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 20044kB

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840970] Node 0 Normal: 568*4kB 89*8kB 60*16kB 1*32kB 1*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 5192kB

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840977] 103770 total pagecache pages

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840979] 102803 pages in swap cache

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840980] Swap cache stats: add 4637080, delete 4534277, find 1317032/1619319

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840981] Free swap  = 0kB

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.840982] Total swap = 5242876kB

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852751] 1968112 pages RAM

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852754] 60735 pages reserved

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852755] 1469 pages shared

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852756] 1897365 pages non-shared

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852758] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852767] [  452]     0   452     4308        0   1       0             0 upstart-udev-br

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852771] [  455]     0   455     5399        1   1     -17         -1000 udevd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852774] [  546]     0   546     5398        1   1     -17         -1000 udevd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852777] [  566]     0   566     5398        1   1     -17         -1000 udevd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852780] [  602]     0   602     3797        0   0       0             0 upstart-socket-

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852783] [  711]     0   711     1816       36   0       0             0 dhclient3

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852788] [  865]     0   865    12489       10   0     -17         -1000 sshd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852791] [  877]   102   877     5979        0   0       0             0 dbus-daemon

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852794] [  880]   101   880    62562      995   0       0             0 rsyslogd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852797] [  946]     0   946     3626        2   0       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852800] [  957]     0   957     3626        2   0       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852803] [  964]     0   964     3626        2   0       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852806] [  967]     0   967     1082        4   0       0             0 runsvdir

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852809] [  969]     0   969     3626        2   0       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852812] [  971]     0   971     3626        2   1       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852815] [  972]     0   972     1044        4   0       0             0 runsv

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852818] [  980]     0   980     1080        5   0       0             0 svlogd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852822] [  987]   108   987    12062       64   0       0             0 snmpd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852825] [  991]     0   991     1082        1   1       0             0 acpid

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852828] [  992]     0   992     4778       38   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852831] [  994]     0   994     4227        1   0       0             0 atd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852834] [ 1024]   103  1024    46896        0   0       0             0 whoopsie

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852837] [ 1049]     0  1049     1202       18   1       0             0 mdadm

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852840] [ 1148]     0  1148     3626        2   0       0             0 getty

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852844] [ 1412]   109  1412     9424       34   0       0             0 ntpd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852847] [21500]   999 21500   276343     3874   0       0             0 sensu-client

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852850] [20206]   107 20206 256226471  1534219   1       0             0 mongod

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852854] [25332]     0 25332     1073       21   1       0             0 collectdmon

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852857] [25333]     0 25333   192855      222   0       0             0 collectd

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852860] [29485]     0 29485     8461       76   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852863] [29495]     0 29495     2753       49   0       0             0 bash

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852866] [29519]     0 29519     1470       22   0       0             0 sleep

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852869] [30761]   999 30761     1100       25   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852872] [30762]   999 30762     7335      985   0       0             0 ruby

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852875] [30764]   999 30764     1100       24   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852878] [30766]   999 30766     7246      888   0       0             0 ruby

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852882] [30768]     0 30768     8461       76   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852885] [30769]     0 30769     8461       76   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852888] [30771]     0 30771     8461       76   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852891] [30773]     0 30773     8461       76   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852894] [30777]  1004 30777     1100       26   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852897] [30778]  1002 30778     1100       26   1       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852900] [30779]  1003 30779     1100       26   1       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852903] [30780]  1001 30780     1100       25   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852906] [30781]  1003 30781     2753       50   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852909] [30782]  1004 30782     2753       50   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852913] [30783]  1002 30783     2753       50   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852916] [30784]  1001 30784     2753       49   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852919] [30825]  1004 30825     3489       59   0       0             0 git

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852922] [30826]  1003 30826     3489       59   1       0             0 git

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852924] [30828]  1002 30828     3490       59   0       0             0 git

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852927] [30829]  1001 30829     3490       59   1       0             0 git

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852930] [30838]  1001 30838    10330      162   0       0             0 ssh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852933] [30839]  1004 30839    10330      162   1       0             0 ssh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852935] [30840]  1003 30840    10330      161   0       0             0 ssh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852938] [30841]  1002 30841    10330      162   1       0             0 ssh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852941] [30843]     0 30843     8461       76   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852944] [30844]     0 30844     8461       76   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852947] [30845]     0 30845     8461       76   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852950] [30846]     0 30846     8460       98   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852953] [30847]     0 30847     8461       76   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852956] [30848]     0 30848     8460       99   1       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852958] [30849]     0 30849     8461       76   0       0             0 cron

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852961] [30850]     0 30850     1100       28   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852964] [30851]  1002 30851     1100       39   1       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852967] [30852]  1001 30852     1100       39   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852970] [30854]  1003 30854     1100       39   0       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852973] [30855]  1004 30855     1100       38   1       0             0 sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852976] [30858]  1004 30858     2349       47   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852979] [30859]  1003 30859     2349       47   1       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852982] [30862]  1002 30862     2349       47   0       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852985] [30863]  1001 30863     2349       47   1       0             0 pull.sh

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852988] [30864]     0 30864     1067       56   0       0             0 debian-sa1

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.852991] Out of memory: Kill process 20206 (mongod) score 948 or sacrifice child

Feb  3 10:05:14 mongo-publicdirectory02n06 kernel: [1103335.853094] Killed process 20206 (mongod) total-vm:1024905884kB, anon-rss:6136876kB, file-rss:0kB

Feb  3 10:05:16 mongo-publicdirectory02n06 kernel: [1103339.229837] init: mongod main process (20206) killed by KILL signal

mongodb.log

2015-02-03T10:04:48.638-0500 [conn2120345] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f7d6e80e82f11690dae3'), member: 14, config: { _id: 14, host: "mongo-publicdirectory02n04.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 336ms

2015-02-03T10:04:49.635-0500 [conn2120341] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 417ms

2015-02-03T10:04:49.635-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 417ms

2015-02-03T10:04:49.635-0500 [conn2120346] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f79abf49a3f368c4f811'), member: 15, config: { _id: 15, host: "mongo-publicdirectory02n05.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 417ms

2015-02-03T10:04:49.635-0500 [TTLMonitor] query admin.system.indexes query: { expireAfterSeconds: { $exists: true } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:1 nscannedObjects:1 keyUpdates:0 numYields:1 locks(micros) r:4128691 nreturned:0 reslen:20 2534ms

2015-02-03T10:04:49.854-0500 [conn2120345] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f79abf49a3f368c4f811'), member: 15, config: { _id: 15, host: "mongo-publicdirectory02n05.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 325ms

2015-02-03T10:04:51.273-0500 [conn2120346] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f79abf49a3f368c4f811'), member: 15, config: { _id: 15, host: "mongo-publicdirectory02n05.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 201ms

2015-02-03T10:04:51.273-0500 [conn2120345] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f7bbe845ad715bb7ad10'), member: 16, config: { _id: 16, host: "mongo-publicdirectory02n06.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 201ms

2015-02-03T10:04:52.093-0500 [TTLMonitor] query config.system.indexes query: { expireAfterSeconds: { $exists: true } } planSummary: EOF ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:0 keyUpdates:0 numYields:0 locks(micros) r:989160 nreturned:0 reslen:20 1099ms

2015-02-03T10:04:53.019-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 208ms

2015-02-03T10:04:53.019-0500 [conn2120341] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 208ms

2015-02-03T10:04:53.019-0500 [conn2120346] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f7bbe845ad715bb7ad10'), member: 16, config: { _id: 16, host: "mongo-publicdirectory02n06.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 208ms

2015-02-03T10:04:54.346-0500 [conn354329] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn409147] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn1305493] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn1133365] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn2118588] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn1484584] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn349345] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn353809] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [conn354267] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 434ms

2015-02-03T10:04:54.346-0500 [slaveTracking] update local.slaves query: { _id: ObjectId('5433f79abf49a3f368c4f811'), config: { _id: 15, host: "mongo-publicdirectory02n05.contactive.com:27017" }, ns: "local.oplog.rs" } update: { $set: { syncedTo: Timestamp 1422975834000|1 } } nscanned:1 nscannedObjects:1 nModified:0 fastmodinsert:1 upsert:1 keyUpdates:0 exception: insertDocument :: caused by :: 11000 E11000 duplicate key error index: local.slaves.$_id_  dup key: { : ObjectId('5433f79abf49a3f368c4f811') } code:11000 numYields:1 locks(micros) w:10006177 10732ms

2015-02-03T10:04:55.241-0500 [conn2114299] command admin.$cmd command: ping { ping: 1 } keyUpdates:0 numYields:0  reslen:37 101ms

2015-02-03T10:04:55.241-0500 [conn2117897] command admin.$cmd command: ping { ping: 1 } keyUpdates:0 numYields:0  reslen:37 101ms

2015-02-03T10:04:55.241-0500 [conn2120346] command admin.$cmd command: replSetUpdatePosition { replSetUpdatePosition: 1, handshake: { handshake: ObjectId('5433f7d6e80e82f11690dae3'), member: 14, config: { _id: 14, host: "mongo-publicdirectory02n04.contactive.com:27017" } } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 233ms

2015-02-03T10:04:55.348-0500 [conn116] command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:1098 451ms

2015-02-03T10:04:55.352-0500 [conn2120300] killcursors  keyUpdates:0 numYields:0 locks(micros) r:659744 8706ms

2015-02-03T10:04:56.349-0500 [initandlisten] connection accepted from 10.0.8.227:59540 #2120347 (770 connections now open)

2015-02-03T10:04:56.354-0500 [conn2114271] update publicDirectoryLog.publicDirectoryLog query: { _id: 273031278731953557 } update: { _id: 273031278731953557, date: new Date(1422975837603), phone: "+27744518036", resolved: false, hasdata: false, justCreated: false, duration: 171, agent: "android-contactive-http/1.12.2", ip: "41.48.16.54", target: "Incoming_Call_Non_User", origins: {}, queries: {}, country: "ZA", r: -1.0, n: 0, maxConfidence: -1, userId: 268179134425636517 } nscanned:0 nscannedObjects:0 nMatched:1 nModified:0 upsert:1 keyUpdates:0 numYields:0 locks(micros) w:11114414 49298ms

2015-02-03T10:04:56.361-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 219ms

2015-02-03T10:04:56.571-0500 [conn2114255] insert publicDirectory.publicDirectory query: { _id: "+52345515408074", keyType: "phone", revision: 1, listed: true, origins: [ { name: { firstName: "Spinelli", normalized: "spinelli", phonetic: [ "SPNL" ] }, originName: "crowdsource", originItemId: "spinelli", confidence: 0, confidenceOperations: [], confidenceBackup: -2147483648, contactType: "single", picture: [], statusUpdate: [], about: "", summary: "", url: [], address: [], work: [], education: [], devices: [], itemReferences: [ { applicationName: "contactive", confidence: 1, weak: false, ownerId: 250142262894495125, itemId: 273035675369178517 } ], sourceNames: [ { sourceName: "crowdsource" } ], email: [], phone: [], event: [], skill: [], honor: [], interest: [], language: [], following: [], patent: [], publication: [], group: [], opportunity: [], note: [], nickname: [] } ], queryHistory: [], created: new Date(1422975837389), targetFrequency: {}, lastModified: new Date(1422975837393), spamScore: 0, spamOperations: [], migrationStatus: 0, migrations: {} } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:5891512 58213ms

2015-02-03T10:04:56.571-0500 [conn2118855] update pictures.pictureCopy query: { originalUrl: "https://graph.facebook.com/1158228189/picture?type=square&width=500&height=500&_t=1422903855000" } update: { $set: { copyUrl: "https://d1j2or3azepuq.cloudfront.net/live/contact-pictures-x5/23/04/ad/29/2304ad299665a77cb617719262a9c54e68a1475d.jpeg", copyDate: new Date(1422975820755), size: 88397, sha1: "6585677f3fb60c1e4ee4b27ab289a2254e6eadc8" } } nscanned:0 nscannedObjects:0 nMatched:1 nModified:0 fastmodinsert:1 upsert:1 keyUpdates:0 numYields:0 locks(micros) w:12718392 75701ms

2015-02-03T10:04:56.788-0500 [conn2114299] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 numYields:0  reslen:531 227ms

2015-02-03T10:04:56.802-0500 [conn2114255] command publicDirectory.$cmd command: insert { insert: "publicDirectory", documents: [ { _id: "+52345515408074", keyType: "phone", revision: 1, listed: true, origins: [ { name: { firstName: "Spinelli", normalized: "spinelli", phonetic: [ "SPNL" ] }, originName: "crowdsource", originItemId: "spinelli", confidence: 0, confidenceOperations: [], confidenceBackup: -2147483648, contactType: "single", picture: [], statusUpdate: [], about: "", summary: "", url: [], address: [], work: [], education: [], devices: [], itemReferences: [ { applicationName: "contactive", confidence: 1, weak: false, ownerId: 250142262894495125, itemId: 273035675369178517 } ], sourceNames: [ { sourceName: "crowdsource" } ], email: [], phone: [], event: [], skill: [], honor: [], interest: [], language: [], following: [], patent: [], publication: [], group: [], opportunity: [], note: [], nickname: [] } ], queryHistory: [], created: new Date(1422975837389), targetFrequency: {}, lastModified: new Date(1422975837393), spamScore: 0, spamOperations: [], migrationStatus: 0, migrations: {} } ], ordered: true, metadata: { shardName: "publicdirectory02", shardVersion: [ Timestamp 1367000|27, ObjectId('5244a7423fbc8427270757f6') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0 locks(micros) w:315686 reslen:140 58990ms

2015-02-03T10:04:56.802-0500 [conn2118855] command pictures.$cmd command: update { update: "pictureCopy", updates: [ { q: { originalUrl: "https://graph.facebook.com/1158228189/picture?type=square&width=500&height=500&_t=1422903855000" }, u: { $set: { copyUrl: "https://d1j2or3azepuq.cloudfront.net/live/contact-pictures-x5/23/04/ad/29/2304ad299665a77cb617719262a9c54e68a1475d.jpeg", copyDate: new Date(1422975820755), size: 88397, sha1: "6585677f3fb60c1e4ee4b27ab289a2254e6eadc8" } }, upsert: true } ], ordered: true, metadata: { shardName: "publicdirectory02", shardVersion: [ Timestamp 151000|259, ObjectId('53fce392aa5413cf4ddfeefc') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:206 76035ms

2015-02-03T10:04:56.812-0500 [conn2114271] command publicDirectoryLog.$cmd command: update { update: "publicDirectoryLog", updates: [ { q: { _id: 273031278731953557 }, u: { _id: 273031278731953557, date: new Date(1422975837603), phone: "+27744518036", resolved: false, hasdata: false, justCreated: false, duration: 171, agent: "android-contactive-http/1.12.2", ip: "41.48.16.54", target: "Incoming_Call_Non_User", origins: {}, queries: {}, country: "ZA", r: -1.0, n: 0, maxConfidence: -1, userId: 268179134425636517 }, multi: false, upsert: true } ], writeConcern: { w: 1 }, ordered: true, metadata: { shardName: "publicdirectory02", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:202 50505ms

2015-02-03T10:04:57.780-0500 [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 25ms

2015-02-03T10:04:57.780-0500 [conn62] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 327ms

2015-02-03T10:04:58.319-0500 [conn2120341] end connection 10.0.8.227:59526 (769 connections now open)

2015-02-03T10:04:58.319-0500 [conn2120300] end connection 10.0.8.227:59396 (769 connections now open)

2015-02-03T10:04:58.853-0500 [conn2118611] query publicDirectory.publicDirectory query: { $query: { _id: "+198633452" } } planSummary: IDHACK ntoskip:0 nscanned:1 nscannedObjects:1 idhack:1 keyUpdates:0 numYields:0 locks(micros) r:1946034 nreturned:1 reslen:2146 1854ms

2015-02-03T10:04:58.853-0500 [conn2114267] query publicDirectory.publicDirectory query: { $query: { _id: "+18777959819" } } planSummary: IDHACK ntoskip:0 nscanned:1 nscannedObjects:1 idhack:1 keyUpdates:0 numYields:0 locks(micros) r:1946070 nreturned:1 reslen:45396 1854ms

2015-02-03T10:04:58.864-0500 [PeriodicTaskRunner] task: WriteBackManager::cleaner took: 88ms

2015-02-03T10:04:59.114-0500 [conn2120347] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 319ms

2015-02-03T10:04:59.225-0500 [conn116] command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:1098 349ms

2015-02-03T10:04:59.225-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 236ms

2015-02-03T10:04:59.325-0500 [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 197ms

2015-02-03T10:04:59.764-0500 [conn116] command admin.$cmd command: replSetGetStatus { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:1098 320ms

2015-02-03T10:04:59.764-0500 [conn2117897] serverStatus was very slow: { after basic: 100, after asserts: 210, after backgroundFlushing: 300, after connections: 310, after cursors: 320, after dur: 430, after extra_info: 750, after globalLock: 760, after indexCounters: 850, after network: 860, after opcounters: 970, after opcountersRepl: 970, after repl: 1220, at end: 1540 }

2015-02-03T10:05:00.761-0500 [conn2117897] command admin.$cmd command: serverStatus { serverStatus: 1, recordStats: 0, locks: 0 } keyUpdates:0 numYields:0  reslen:3145 2434ms

2015-02-03T10:05:02.373-0500 [conn2120347] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 343ms

2015-02-03T10:05:02.373-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 343ms

2015-02-03T10:05:02.373-0500 [conn2117897] command admin.$cmd command: buildInfo { buildinfo: 1 } keyUpdates:0 numYields:0  reslen:699 343ms

2015-02-03T10:05:02.497-0500 [conn2114245] query publicDirectory.publicDirectory query: { $query: { _id: "+34655778473" } } planSummary: IDHACK ntoskip:0 nscanned:1 nscannedObjects:1 idhack:1 keyUpdates:0 numYields:0 locks(micros) r:1517230 nreturned:1 reslen:8877 1606ms

2015-02-03T10:05:03.673-0500 [conn2117897] command admin.$cmd command: getCmdLineOpts { getCmdLineOpts: 1 } keyUpdates:0 numYields:0  reslen:406 147ms

2015-02-03T10:05:04.537-0500 [conn2114657] update publicDirectoryLog.publicDirectoryLog query: { _id: 149298287690876309 } update: { _id: 149298287690876309, date: new Date(1422975851685), phone: "+13306681400", resolved: true, name: "Mark Rodio", hasdata: true, justCreated: false, duration: 2073, agent: "android-klink-http/2.2.1", ip: "70.194.202.49", target: "NA", origins: { facebook: 2, crowdsource: 1 }, queries: {}, country: "US", r: 0.6666666865348816, n: 3, maxConfidence: 2, userId: 195941630968293909 } nscanned:0 nscannedObjects:0 nMatched:1 nModified:0 upsert:1 keyUpdates:0 numYields:0 locks(micros) w:8281033 50614ms

2015-02-03T10:05:05.340-0500 [conn2120347] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 319ms

2015-02-03T10:05:05.340-0500 [conn2117897] command admin.$cmd command: connPoolStats { connPoolStats: 1 } keyUpdates:0 numYields:0  reslen:818 331ms

2015-02-03T10:05:05.340-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 319ms

2015-02-03T10:05:05.670-0500 [conn409147] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn353809] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn1484584] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn1305493] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn1133365] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn354267] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn354329] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn2118588] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.670-0500 [conn349345] command admin.$cmd command: isMaster { ismaster: 1 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:531 218ms

2015-02-03T10:05:05.783-0500 [conn2119780] update pictures.pictureCopy query: { originalUrl: "https://graph.facebook.com/602969410/picture?type=square&width=500&height=500&_t=1422924074000" } update: { $set: { copyUrl: "https://d1j2or3azepuq.cloudfront.net/live/contact-pictures-x5/e7/22/cd/36/e722cd3661c9cf477c3c7bba9fc4f5c38de85850.jpeg", copyDate: new Date(1422975820830), size: 16600, sha1: "7c93ff70591056076b9fcb8e14017ddcddd93169" } } nscanned:0 nscannedObjects:0 nMatched:1 nModified:0 fastmodinsert:1 upsert:1 keyUpdates:0 numYields:0 locks(micros) w:9523220 84837ms

2015-02-03T10:05:05.783-0500 [conn2114657] command publicDirectoryLog.$cmd command: update { update: "publicDirectoryLog", updates: [ { q: { _id: 149298287690876309 }, u: { _id: 149298287690876309, date: new Date(1422975851685), phone: "+13306681400", resolved: true, name: "Mark Rodio", hasdata: true, justCreated: false, duration: 2073, agent: "android-klink-http/2.2.1", ip: "70.194.202.49", target: "NA", origins: { facebook: 2, crowdsource: 1 }, queries: {}, country: "US", r: 0.6666666865348816, n: 3, maxConfidence: 2, userId: 195941630968293909 }, multi: false, upsert: true } ], writeConcern: { w: 1 }, ordered: true, metadata: { shardName: "publicdirectory02", shardVersion: [ Timestamp 0|0, ObjectId('000000000000000000000000') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:202 52613ms

2015-02-03T10:05:07.855-0500 [conn2117897] command admin.$cmd command: getLog { getLog: "startupWarnings" } keyUpdates:0 numYields:0  reslen:70 105ms

2015-02-03T10:05:07.855-0500 [conn2119780] command pictures.$cmd command: update { update: "pictureCopy", updates: [ { q: { originalUrl: "https://graph.facebook.com/602969410/picture?type=square&width=500&height=500&_t=1422924074000" }, u: { $set: { copyUrl: "https://d1j2or3azepuq.cloudfront.net/live/contact-pictures-x5/e7/22/cd/36/e722cd3661c9cf477c3c7bba9fc4f5c38de85850.jpeg", copyDate: new Date(1422975820830), size: 16600, sha1: "7c93ff70591056076b9fcb8e14017ddcddd93169" } }, upsert: true } ], ordered: true, metadata: { shardName: "publicdirectory02", shardVersion: [ Timestamp 151000|259, ObjectId('53fce392aa5413cf4ddfeefc') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0  reslen:206 86692ms

2015-02-03T10:05:07.855-0500 [conn2114299] command admin.$cmd command: ping { ping: 1 } keyUpdates:0 numYields:0  reslen:37 110ms

2015-02-03T10:05:10.398-0500 [conn2120347] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n04.contactive.com:27017", fromId: 14 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 550ms

2015-02-03T10:05:10.398-0500 [conn2120342] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "publicdirectory02", v: 66274, pv: 1, checkEmpty: false, from: "mongo-publicdirectory02n05.contactive.com:27017", fromId: 15 } ntoreturn:1 keyUpdates:0 numYields:0  reslen:155 432ms

2015-02-03T10:05:10.398-0500 [conn2117897] command admin.$cmd command: hostInfo { hostInfo: 1 } keyUpdates:0 numYields:0  reslen:893 220ms

2015-02-03T10:17:26.391-0500 ***** SERVER RESTARTED *****

All help and advice is greatly appreciated.