On Tue, Feb 22, 2011 at 8:46 PM, Avery Fay <av...@shadypixel.com> wrote:
> Hi,
> Last week we added new shard to our cluster, and one of the existing
> mongod's (shardsrv) promptly crashed. After a very long repair, everything
> started working again (additionally, we've added yet another shard since
> with no crash), but I figured I'd post the backtrace anyway.
> Avery
> End of log:
> Sat Feb 19 03:33:15 [conn2022] MessagingPort say send() errno:32 Broken pipe
> 10.177.200.233:27019
> Sat Feb 19 03:33:15 [conn2022] MessagingPort say send() errno:32 Broken pipe
> 10.177.200.233:27019
> Sat Feb 19 03:33:15 terminate() called, printing stack:
> 0x823830 0x7ffa45b85396 0x7ffa45b8458b 0x7ffa45b85158 0x7ffa45631ff3
> 0x7ffa456320b8 0x807051 0x80fbe1 0x797117 0x798538 0x5fb7e5 0x60029f
> 0x7074ba 0x70aaf6 0x82691b 0x83a4b0 0x7ffa45dd23ea 0x7ffa45395cbd
>  bin/mongod(_ZN5mongo11myterminateEv+0x50) [0x823830]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b85396]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
>  /usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
>  /lib/libgcc_s.so.1 [0x7ffa45631ff3]
>  /lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
>  bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
>  bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONOb jBuilderEb+0x4ee1)
> [0x80fbe1]
>  bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObj ERNS_14BSONObjBuilderEb+0x597)
> [0x797117]
>  bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14B SONObjBuilderEbi+0x798)
> [0x798538]
>  bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuild erERNS_14BSONObjBuilderEbi+0x35)
> [0x5fb7e5]
>  bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 _+0x1bbf)
> [0x60029f]
>  bin/mongod [0x7074ba]
>  bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8 SockAddrE+0x14d6)
> [0x70aaf6]
>  bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x30b) [0x82691b]
>  bin/mongod(thread_proxy+0x80) [0x83a4b0]
>  /lib/libpthread.so.0 [0x7ffa45dd23ea]
>  /lib/libc.so.6(clone+0x6d) [0x7ffa45395cbd]
> Sat Feb 19 03:33:16 Got signal: 6 (Aborted).
> Sat Feb 19 03:33:16 Backtrace:
> 0x824629 0x7ffa452e20a0 0x7ffa452e2015 0x7ffa452e3b83 0x8238f7
> 0x7ffa45b85396 0x7ffa45b8458b 0x7ffa45b85158 0x7ffa45631ff3 0x7ffa456320b8
> 0x807051 0x80fbe1 0x797117 0x798538 0x5fb7e5 0x60029f 0x7074ba 0x70aaf6
> 0x82691b 0x83a4b0
>  bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x824629]
>  /lib/libc.so.6 [0x7ffa452e20a0]
>  /lib/libc.so.6(gsignal+0x35) [0x7ffa452e2015]
>  /lib/libc.so.6(abort+0x183) [0x7ffa452e3b83]
>  bin/mongod(_ZN5mongo11myterminateEv+0x117) [0x8238f7]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b85396]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
>  /usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
>  /lib/libgcc_s.so.1 [0x7ffa45631ff3]
>  /lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
>  bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
>  bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONOb jBuilderEb+0x4ee1)
> [0x80fbe1]
>  bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObj ERNS_14BSONObjBuilderEb+0x597)
> [0x797117]
>  bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14B SONObjBuilderEbi+0x798)
> [0x798538]
>  bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuild erERNS_14BSONObjBuilderEbi+0x35)
> [0x5fb7e5]
>  bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 _+0x1bbf)
> [0x60029f]
>  bin/mongod [0x7074ba]
>  bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8 SockAddrE+0x14d6)
> [0x70aaf6]
>  bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x30b) [0x82691b]
>  bin/mongod(thread_proxy+0x80) [0x83a4b0]
> Sat Feb 19 03:33:16 dbexit:
> Sat Feb 19 03:33:16 [conn2022] shutdown: going to close listening sockets...
> Sat Feb 19 03:33:16 [conn2022] closing listening socket: 5
> Sat Feb 19 03:33:16 [conn2022] closing listening socket: 6
> Sat Feb 19 03:33:16 [conn2022] closing listening socket: 7
> Sat Feb 19 03:33:16 [conn2022] closing listening socket: 8
> Sat Feb 19 03:33:16 [conn2022] shutdown: going to flush oplog...
> Sat Feb 19 03:33:16 [conn2022] shutdown: going to close sockets...
> Sat Feb 19 03:33:16 [conn2022] shutdown: waiting for fs preallocator...
> Sat Feb 19 03:33:16 [conn2022] shutdown: closing all files...
> Sat Feb 19 03:33:16 [initandlisten] now exiting
> Sat Feb 19 03:33:16 dbexit: ; exiting immediately
> Sat Feb 19 03:33:16 Got signal: 11 (Segmentation fault).
> Sat Feb 19 03:33:16 Backtrace:
> 0x824629 0x7ffa452e20a0 0x52c2b5 0x701d57 0x702551 0x824773 0x7ffa452e20a0
> 0x7ffa452e2015 0x7ffa452e3b83 0x8238f7 0x7ffa45b85396 0x7ffa45b8458b
> 0x7ffa45b85158 0x7ffa45631ff3 0x7ffa456320b8 0x807051 0x80fbe1 0x797117
> 0x798538 0x5fb7e5
>  bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x824629]
>  /lib/libc.so.6 [0x7ffa452e20a0]
>  bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11 char_traitsIcESaIcEE+0xa5)
> [0x52c2b5]
>  bin/mongod(_ZN5mongo8shutdownEv+0x3a7) [0x701d57]
>  bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKc+0x201) [0x702551]
>  bin/mongod(_ZN5mongo10abruptQuitEi+0x4e3) [0x824773]
>  /lib/libc.so.6 [0x7ffa452e20a0]
>  /lib/libc.so.6(gsignal+0x35) [0x7ffa452e2015]
>  /lib/libc.so.6(abort+0x183) [0x7ffa452e3b83]
>  bin/mongod(_ZN5mongo11myterminateEv+0x117) [0x8238f7]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b85396]
>  /usr/lib/libstdc++.so.6 [0x7ffa45b8458b]
>  /usr/lib/libstdc++.so.6(__gxx_personality_v0+0x358) [0x7ffa45b85158]
>  /lib/libgcc_s.so.1 [0x7ffa45631ff3]
>  /lib/libgcc_s.so.1(_Unwind_Resume+0x68) [0x7ffa456320b8]
>  bin/mongod(_ZN5mongo16MoveTimingHelperD1Ev+0x2b1) [0x807051]
>  bin/mongod(_ZN5mongo16MoveChunkCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONOb jBuilderEb+0x4ee1)
> [0x80fbe1]
>  bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObj ERNS_14BSONObjBuilderEb+0x597)
> [0x797117]
>  bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14B SONObjBuilderEbi+0x798)
> [0x798538]
>  bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuild erERNS_14BSONObjBuilderEbi+0x35)
> [0x5fb7e5]

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.