mongod crash when new shard is added
48 views
Skip to first unread message
Avery Fay
Feb 22, 2011, 8:46:28 PM2/22/11
to mongod...@googlegroups.com
Hi,
Last week we added new shard to our cluster, and one of the existing mongod's (shardsrv) promptly crashed. After a very long repair, everything started working again (additionally, we've added yet another shard since with no crash), but I figured I'd post the backtrace anyway.
Avery
End of log:
Sat Feb 19 03:33:15 [conn2022] MessagingPort say send() errno:32 Broken pipe 10.177.200.233:27019
Sat Feb 19 03:33:15 [conn2022] MessagingPort say send() errno:32 Broken pipe 10.177.200.233:27019
Sat Feb 19 03:33:15 terminate() called, printing stack:
0x823830 0x7ffa45b85396 0x7ffa45b8458b 0x7ffa45b85158 0x7ffa45631ff3 0x7ffa456320b8 0x807051 0x80fbe1 0x797117 0x798538 0x5fb7e5 0x60029f 0x7074ba 0x70aaf6 0x82691b 0x83a4b0 0x7ffa45dd23ea 0x7ffa45395cbd
bin/mongod(_ZN5mongo11myterminateEv+0x50) [0x823830]
/usr/lib/libstdc++.so.6 [0x7ffa45b85396]
/usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
/usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
/lib/libgcc_s.so.1 [0x7ffa45631ff3]
/lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x4ee1) [0x80fbe1]
bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x597) [0x797117]
bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x798) [0x798538]
bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x5fb7e5]
bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1bbf) [0x60029f]
bin/mongod [0x7074ba]
bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x14d6) [0x70aaf6]
bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x30b) [0x82691b]
bin/mongod(thread_proxy+0x80) [0x83a4b0]
/lib/libpthread.so.0 [0x7ffa45dd23ea]
/lib/libc.so.6(clone+0x6d) [0x7ffa45395cbd]
Sat Feb 19 03:33:16 Got signal: 6 (Aborted).
Sat Feb 19 03:33:16 Backtrace:
0x824629 0x7ffa452e20a0 0x7ffa452e2015 0x7ffa452e3b83 0x8238f7 0x7ffa45b85396 0x7ffa45b8458b 0x7ffa45b85158 0x7ffa45631ff3 0x7ffa456320b8 0x807051 0x80fbe1 0x797117 0x798538 0x5fb7e5 0x60029f 0x7074ba 0x70aaf6 0x82691b 0x83a4b0
bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x824629]
/lib/libc.so.6 [0x7ffa452e20a0]
/lib/libc.so.6(gsignal+0x35) [0x7ffa452e2015]
/lib/libc.so.6(abort+0x183) [0x7ffa452e3b83]
bin/mongod(_ZN5mongo11myterminateEv+0x117) [0x8238f7]
/usr/lib/libstdc++.so.6 [0x7ffa45b85396]
/usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
/usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
/lib/libgcc_s.so.1 [0x7ffa45631ff3]
/lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x4ee1) [0x80fbe1]
bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x597) [0x797117]
bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x798) [0x798538]
bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x5fb7e5]
bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1bbf) [0x60029f]
bin/mongod [0x7074ba]
bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x14d6) [0x70aaf6]
bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x30b) [0x82691b]
bin/mongod(thread_proxy+0x80) [0x83a4b0]
Sat Feb 19 03:33:16 dbexit:
Sat Feb 19 03:33:16 [conn2022] shutdown: going to close listening sockets...
Sat Feb 19 03:33:16 [conn2022] closing listening socket: 5
Sat Feb 19 03:33:16 [conn2022] closing listening socket: 6
Sat Feb 19 03:33:16 [conn2022] closing listening socket: 7
Sat Feb 19 03:33:16 [conn2022] closing listening socket: 8
Sat Feb 19 03:33:16 [conn2022] shutdown: going to flush oplog...
Sat Feb 19 03:33:16 [conn2022] shutdown: going to close sockets...
Sat Feb 19 03:33:16 [conn2022] shutdown: waiting for fs preallocator...
Sat Feb 19 03:33:16 [conn2022] shutdown: closing all files...
Sat Feb 19 03:33:16 [initandlisten] now exiting
Sat Feb 19 03:33:16 dbexit: ; exiting immediately
Sat Feb 19 03:33:16 Got signal: 11 (Segmentation fault).
Sat Feb 19 03:33:16 Backtrace:
0x824629 0x7ffa452e20a0 0x52c2b5 0x701d57 0x702551 0x824773 0x7ffa452e20a0 0x7ffa452e2015 0x7ffa452e3b83 0x8238f7 0x7ffa45b85396 0x7ffa45b8458b 0x7ffa45b85158 0x7ffa45631ff3 0x7ffa456320b8 0x807051 0x80fbe1 0x797117 0x798538 0x5fb7e5
bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x824629]
/lib/libc.so.6 [0x7ffa452e20a0]
bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0xa5) [0x52c2b5]
bin/mongod(_ZN5mongo8shutdownEv+0x3a7) [0x701d57]
bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKc+0x201) [0x702551]
bin/mongod(_ZN5mongo10abruptQuitEi+0x4e3) [0x824773]
/lib/libc.so.6 [0x7ffa452e20a0]
/lib/libc.so.6(gsignal+0x35) [0x7ffa452e2015]
/lib/libc.so.6(abort+0x183) [0x7ffa452e3b83]
bin/mongod(_ZN5mongo11myterminateEv+0x117) [0x8238f7]
/usr/lib/libstdc++.so.6 [0x7ffa45b85396]
/usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
/usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
/lib/libgcc_s.so.1 [0x7ffa45631ff3]
/lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x4ee1) [0x80fbe1]
bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x597) [0x797117]
bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x798) [0x798538]
bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x5fb7e5]
Eliot Horowitz
Feb 23, 2011, 5:31:21 AM2/23/11
to mongod...@googlegroups.com
This is an issue fixed in 1.7/1.8 already.
It can happen if there is a network blip between mongod and config
server, or if a config server goes down.
It can happen if there is a network blip between mongod and config
server, or if a config server goes down.
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
Reply all
Reply to author
Forward
0 new messages