MongoS crashing with 9001 socket exception

[Darshan Shah]

I am trying to start a MongoS process on an old bix running CentOS5.2 and it starts fine.

The first client also connects fine, however as soon as a second client connects to this MongoS, it crashed with the below error message in the log:

2016-05-09T15:44:17.648-0400 D NETWORK  [conn1] Socket recv() conn closed? 127.0.0.1:22228

2016-05-09T15:44:17.648-0400 F -        [conn1] terminate() called. An exception is active; attempting to gather more information

2016-05-09T15:44:17.654-0400 F -        [conn1] DBException::toString(): 9001 socket exception [CLOSED] server [127.0.0.1:22228]

Actual exception type: mongo::SocketException

 0xc6e362 0xc6dc82 0xe5dcc6 0xe5dcf3 0xe5ddfe 0xc20ee1 0xc2227b 0x21d8480

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"86E362","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"86DC82"},{"b":"400000","o":"A5DCC6","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"400000","o":"A5DCF3"},{"b":"400000","o":"A5DDFE"},{"b":"400000","o":"820EE1","s":"_ZN5mongo6Socket15handleRecvErrorEii"},{"b":"400000","o":"82227B","s":"_ZN5mongo6Socket5_recvEPci"},{"b":"0","o":"21D8480"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.18-274.7.1.el5", "version" : "#1 SMP Mon Oct 17 11:57:14 EDT 2011", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFCE8FD000", "elfType" : 3 }, { "path" : "/lib64/libssl.so.6", "elfType" : 3 }, { "path" : "/lib64/libcrypto.so.6", "elfType" : 3 }, { "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3 }, { "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3 }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3 }, { "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3 }, { "path" : "/usr/lib64/libz.so.1", "elfType" : 3 }, { "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3 }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3 }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3 }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3 }, { "path" : "/lib64/libsepol.so.1", "elfType" : 3 } ] }}

 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc6e362]

 mongos(+0x86DC82) [0xc6dc82]

 mongos(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0xe5dcc6]

 mongos(+0xA5DCF3) [0xe5dcf3]

 mongos(+0xA5DDFE) [0xe5ddfe]

 mongos(_ZN5mongo6Socket15handleRecvErrorEii+0x401) [0xc20ee1]

 mongos(_ZN5mongo6Socket5_recvEPci+0x6B) [0xc2227b]

 ??? [0x21d8480]

-----  END BACKTRACE  -----

Here are the limits setup on this box:

-bash-3.2$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 593919

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) unlimited

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

Here are the contents of the file /etc/security/limits.conf showing that limits should not be a problem:

oasprod soft nofile unlimited

oasprod hard nofile 65535

oasprod soft nproc unlimited

oasprod hard nproc unlimited

oasprod soft fsize unlimited

oasprod hard fsize unlimited

oasprod soft memlock unlimited

oasprod hard memlock unlimited

oasprod soft cpu unlimited

oasprod hard cpu unlimited

oasprod soft as unlimited

oasprod hard as unlimited

Please let me know any clues as to why this might be happening.

Thanks!

[Darshan Shah]

Missed to mention that this is with MongoDb 3.2.5 with WiredTiger.

[Dharmesh Vyas]

Were you able to resolve this issue.