1.5.7 mongos is crashing often

14 views
Skip to first unread message

breadfan

unread,
Aug 4, 2010, 5:45:49 AM8/4/10
to mongodb-user
Hi, mongos is crashing very often if a lot of UserExceptions happen:

connection with mongo shell I can see:

terminate called after throwing an instance of 'mongo::UserException'
what(): DBClientBase::findOne: transport error
Wed Aug 4 11:44:10 mongo got signal 6 (Abort trap), stack trace:

Wed Aug 4 11:44:10 0x27be 0x96fa61fb 0xffffffff 0x9703362d
0x970496e4 0x98e9cfda 0x98e9b17a 0x98e9b1ba 0x98e9b2b8 0xab2cb 0xce119
0xda6a7 0x25851 0x11a67f 0x19bff3 0x19d80f 0x1a7877 0x19c9c9 0x15dd5d
0x15dcb4
0 mongo 0x000027be _Z12quitAbruptlyi
+ 382
1 libSystem.B.dylib 0x96fa61fb _sigtramp + 43
2 ??? 0xffffffff 0x0 + 4294967295
3 libSystem.B.dylib 0x9703362d raise + 26
4 libSystem.B.dylib 0x970496e4 abort + 93
5 libstdc++.6.dylib 0x98e9cfda
_ZN9__gnu_cxx27__verbose_terminate_handlerEv + 433
6 libstdc++.6.dylib 0x98e9b17a
_ZN10__cxxabiv111__terminateEPFvvE + 10
7 libstdc++.6.dylib 0x98e9b1ba
_ZN10__cxxabiv112__unexpectedEPFvvE + 0
8 libstdc++.6.dylib 0x98e9b2b8
_ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception
+ 0
9 mongo 0x000ab2cb
_ZN5mongo9uassertedEiPKc + 283
10 mongo 0x000ce119
_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi +
921
11 mongo 0x000da6a7
_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i +
119
12 mongo 0x00025851
_ZN5mongo10shellUtils9onConnectERNS_20DBClientWithCommandsE + 497
13 mongo 0x0011a67f
_ZN5mongo26mongo_external_constructorEP9JSContextP8JSObjectjPlS4_ +
431
14 mongo 0x0019bff3 js_Invoke + 3275
15 mongo 0x0019d80f
js_InvokeConstructor + 702
16 mongo 0x001a7877 js_Interpret +
40685
17 mongo 0x0019c9c9 js_Execute + 774
18 mongo 0x0015dd5d
JS_EvaluateUCScriptForPrincipals + 167
19 mongo 0x0015dcb4
JS_EvaluateUCScript + 67


Best regards
Chris

breadfan

unread,
Aug 4, 2010, 6:05:49 AM8/4/10
to mongodb-user
There's nothing in the logs but on the console when it dies the
following is printed out:

/lib64/libpthread.so.0 [0x30c1606367]
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x51a3ef 0x54b642 0x54e406 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148)
[0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f)
[0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb)
[0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/
mongos(_ZN5mongo16DBConnectionPool3getERKNS_16ConnectionStringE+0xcf)
[0x51a3ef]
/opt/mongo/bin/
mongos(_ZN5mongo18distLockPingThreadENS_16ConnectionStringE+0x52)
[0x54b642]
/opt/mongo/bin/
mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvPFvN5mongo16ConnectionStringEENS2_5list1INS2_5valueIS5_EEEEEEE3runEv
+0x26)
[0x54e406]
/opt/mongo/bin/mongos(thread_proxy+0x80) [0x66be20]
/lib64/libpthread.so.0 [0x30c1606367]
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x51a3ef 0x6590d9 0x4f1265 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148)
[0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f)
[0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb)
[0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/
mongos(_ZN5mongo16DBConnectionPool3getERKNS_16ConnectionStringE+0xcf)
[0x51a3ef]
/opt/mongo/bin/mongos(_ZN5mongo8Balancer3runEv+0x349) [0x6590d9]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob3thrEv+0x85) [0x4f1265]
/opt/mongo/bin/mongos(thread_proxy+0x80) [0x66be20]
/lib64/libpthread.so.0 [0x30c1606367]

Eliot Horowitz

unread,
Aug 4, 2010, 7:47:10 AM8/4/10
to mongod...@googlegroups.com
There should be something right above that in the logs. Can you send the whole thing?

Also can you try 1.5.8

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

breadfan

unread,
Aug 4, 2010, 8:15:50 AM8/4/10
to mongodb-user
Ok, here's the whole output:

[root@lo24-sv-286 ~]# 0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x5187c6 0x552f20 0x550e68 0x5510a5 0x602e59 0x60e7be 0x63cb89
0x647a29 0x55ad12 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148) [0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f) [0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb) [0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0xd6)
[0x5187c6]
/opt/mongo/bin/mongos(_ZN5mongo17ClientConnections3getERKSs+0x320)
[0x552f20]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x48)
[0x550e68]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs
+0x65) [0x5510a5]
/opt/mongo/bin/
mongos(_ZN5mongo8Strategy7doQueryERNS_7RequestERKNS_5ShardE+0x49)
[0x602e59]
/opt/mongo/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
+0x56e) [0x60e7be]
/opt/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x269) [0x63cb89]
/opt/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x149) [0x647a29]
/opt/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x252) [0x55ad12]
/opt/mongo/bin/mongos(thread_proxy+0x80) [0x66be20]
/lib64/libpthread.so.0 [0x30c1606367]
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x5187c6 0x552f20 0x550e68 0x5510a5 0x602e59 0x60e7be 0x63cb89
0x647a29 0x55ad12 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148) [0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f) [0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb) [0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0xd6)
[0x5187c6]
/opt/mongo/bin/mongos(_ZN5mongo17ClientConnections3getERKSs+0x320)
[0x552f20]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x48)
[0x550e68]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs
+0x65) [0x5510a5]
/opt/mongo/bin/
mongos(_ZN5mongo8Strategy7doQueryERNS_7RequestERKNS_5ShardE+0x49)
[0x602e59]
/opt/mongo/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
+0x56e) [0x60e7be]
/opt/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x269) [0x63cb89]
/opt/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x149) [0x647a29]
/opt/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x252) [0x55ad12]
/opt/mongo/bin/mongos(thread_proxy+0x80) [0x66be20]
/lib64/libpthread.so.0 [0x30c1606367]
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x5187c6 0x552f20 0x550e68 0x5510a5 0x60374e 0x60c394 0x63ca8b
0x647a29 0x55ad12 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148) [0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f) [0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb) [0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0xd6)
[0x5187c6]
/opt/mongo/bin/mongos(_ZN5mongo17ClientConnections3getERKSs+0x320)
[0x552f20]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x48)
[0x550e68]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs
+0x65) [0x5510a5]
/opt/mongo/bin/
mongos(_ZN5mongo8Strategy7doWriteEiRNS_7RequestERKNS_5ShardEb+0x4e)
[0x60374e]
/opt/mongo/bin/mongos(_ZN5mongo14SingleStrategy7writeOpEiRNS_7RequestE
+0xc4) [0x60c394]
/opt/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x16b) [0x63ca8b]
/opt/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x149) [0x647a29]
/opt/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x252) [0x55ad12]
0x5187c6 0x552f20 0x550e68 0x5510a5 0x60374e 0x60c394 0x63ca8b
0x647a29 0x55ad12 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148) [0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f) [0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb) [0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0xd6)
[0x5187c6]
/opt/mongo/bin/mongos(_ZN5mongo17ClientConnections3getERKSs+0x320)
[0x552f20]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x48)
[0x550e68]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs
+0x65) [0x5510a5]
/opt/mongo/bin/
mongos(_ZN5mongo8Strategy7doWriteEiRNS_7RequestERKNS_5ShardEb+0x4e)
[0x60374e]
/opt/mongo/bin/mongos(_ZN5mongo14SingleStrategy7writeOpEiRNS_7RequestE
+0xc4) [0x60c394]
/opt/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x16b) [0x63ca8b]
/opt/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x149) [0x647a29]
/opt/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x252) [0x55ad12]
/lib64/libc.so.6(clone+0x6d) [0x30c0ed309d]
0x4fbc43 0x508401 0x4f0c2e 0x4fd558 0x520c6f 0x520f6b 0x528315
0x5187c6 0x552f20 0x550e68 0x5510a5 0x602e59 0x60e7be 0x63cb89
0x647a29 0x55ad12 0x66be20 0x30c1606367 0x30c0ed309d
/opt/mongo/bin/mongos(_ZN5mongo12sayDbContextEPKc+0xb3) [0x4fbc43]
/opt/mongo/bin/mongos(_ZN5mongo8assertedEPKcS1_j+0x111) [0x508401]
/opt/mongo/bin/mongos(_ZN5mongo13BackgroundJob2goEv+0x4e) [0x4f0c2e]
/opt/mongo/bin/mongos(_ZN5mongo13MessagingPort7connectERNS_8SockAddrE
+0x148) [0x4fd558]
/opt/mongo/bin/mongos(_ZN5mongo18DBClientConnection8_connectERSs
+0x22f) [0x520c6f]
/opt/mongo/bin/
mongos(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs
+0xbb) [0x520f6b]
/opt/mongo/bin/mongos(_ZNK5mongo16ConnectionString7connectERSs+0x2b5)
[0x528315]
/opt/mongo/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0xd6)
[0x5187c6]
/opt/mongo/bin/mongos(_ZN5mongo17ClientConnections3getERKSs+0x320)
[0x552f20]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x48)
[0x550e68]
/opt/mongo/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs
+0x65) [0x5510a5]
/opt/mongo/bin/
mongos(_ZN5mongo8Strategy7doQueryERNS_7RequestERKNS_5ShardE+0x49)
[0x602e59]
/opt/mongo/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
+0x56e) [0x60e7be]
/opt/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x269) [0x63cb89]
/opt/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x149) [0x647a29]
/opt/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x252) [0x55ad12]

breadfan

unread,
Aug 4, 2010, 8:24:04 AM8/4/10
to mongodb-user
We tried the 1.5.8 with static libs and it doesn't crash but it stops
to work correctly after some minutes.

It stops to log new connections etc. and doesn't route new connections
to the mongod shards.

We have two big machines (32 GB Ram, 10 fast disks), on each is a
mongod shard running the mongos runs on first shard server, the config
server runs on the second shard server. They run on CentOS 5.3

breadfan

unread,
Aug 4, 2010, 8:52:43 AM8/4/10
to mongodb-user
Using mongostat on the mongos I see a rising connection count.
Currently 19000 connections...

We have 80 Tomcats each one is allowed to open 75 connections to the
mongos. Should be far below 10.000 connections

Eliot Horowitz

unread,
Aug 4, 2010, 9:51:25 AM8/4/10
to mongod...@googlegroups.com
Its possible the static version is odd - linux generally doesn't like
fully statically linked binaries.
The regular ones don't work.

Could you try master of the java driver out of curiosity?

breadfan

unread,
Aug 4, 2010, 10:22:26 AM8/4/10
to mongodb-user

1.5.8 crashed again now with

Wed Aug 4 16:18:03 connection accepted from 192.168.20.50:38475
#26493
Wed Aug 4 16:18:03 connection accepted from 192.168.20.50:38476
#26494
Wed Aug 4 16:18:03 connection accepted from 192.168.20.50:38477
#26495
Wed Aug 4 16:18:03 [conn24111] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46794
Wed Aug 4 16:18:03 [conn24111] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn24111] UserException: socket exception
Wed Aug 4 16:18:03 [conn24111] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46794
Wed Aug 4 16:18:03 [conn24111] unclean socket shutdown from:
192.168.20.58:46794
Wed Aug 4 16:18:03 [conn26059] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46918
Wed Aug 4 16:18:03 [conn26059] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn26059] UserException: socket exception
Wed Aug 4 16:18:03 [conn26059] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46918
Wed Aug 4 16:18:03 [conn26059] unclean socket shutdown from:
192.168.20.58:46918
Wed Aug 4 16:18:03 connection accepted from 192.168.20.63:54751
#26496
Wed Aug 4 16:18:03 [conn21789] MessagingPort say send() errno:32
Broken pipe 192.168.20.143:51450
Wed Aug 4 16:18:03 [conn21789] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn21789] UserException: socket exception
Wed Aug 4 16:18:03 [conn21789] MessagingPort say send() errno:32
Broken pipe 192.168.20.143:51450
Wed Aug 4 16:18:03 [conn21789] unclean socket shutdown from:
192.168.20.143:51450
Wed Aug 4 16:18:03 [conn19275] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn23158] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46740
Wed Aug 4 16:18:03 [conn23158] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn23158] UserException: socket exception
Wed Aug 4 16:18:03 [conn23158] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46740
Wed Aug 4 16:18:03 [conn23158] unclean socket shutdown from:
192.168.20.58:46740
Wed Aug 4 16:18:03 [conn25802] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46906
Wed Aug 4 16:18:03 [conn25802] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn25802] UserException: socket exception
Wed Aug 4 16:18:03 [conn25802] MessagingPort say send() errno:32
Broken pipe 192.168.20.58:46906
Wed Aug 4 16:18:03 [conn25802] unclean socket shutdown from:
192.168.20.58:46906
Wed Aug 4 16:18:03 [conn22209] MessagingPort say send() errno:32
Broken pipe 192.168.21.56:37910
Wed Aug 4 16:18:03 [conn22209] ~ScopedDBConnection: _conn != null
Wed Aug 4 16:18:03 [conn22209] UserException: socket exception
Wed Aug 4 16:18:03 [conn22209] MessagingPort say send() errno:32
Broken pipe 192.168.21.56:37910
Wed Aug 4 16:18:03 [conn22209] unclean socket shutdown from:
192.168.21.56:37910

Does this help?

Eliot Horowitz

unread,
Aug 4, 2010, 10:24:08 AM8/4/10
to mongod...@googlegroups.com
That's not really a crash - just a lot of socket errors...

Did a client crash or go away?

Could you try newer java driver and/or non static mongo?

breadfan

unread,
Aug 4, 2010, 10:31:39 AM8/4/10
to mongodb-user

We need a redeployment for the new java driver. Not sure if we can do
this today. Do you think the java driver can really bring down the
mongos?

The mongos definitely crashed. We now try the non static mongo again.

breadfan

unread,
Aug 4, 2010, 12:50:14 PM8/4/10
to mongodb-user
The 1.5.8, with dynamically linked libs, crashes with seg faults
unfortunately probably cos of too many connections. Our file handle
limit is high enough (about 65000)

Eliot Horowitz

unread,
Aug 4, 2010, 12:51:07 PM8/4/10
to mongod...@googlegroups.com
Can you paste those segfaults

breadfan

unread,
Aug 5, 2010, 4:26:36 AM8/5/10
to mongodb-user
The only thing we have seen is this:

Aug 4 11:38:41 lo24-sv-286 kernel: mongos[12045]: segfault at
00000000fffffffe rip 0000000000517c10 rsp 0000000042d98620 error 4

breadfan

unread,
Aug 9, 2010, 3:57:03 AM8/9/10
to mongodb-user
We now use the current java driver built from master and use mongos
(1.6.0) on each appserver (about 80) and after some hours of traffic
half of them crashed.

breadfan

unread,
Aug 9, 2010, 4:45:39 AM8/9/10
to mongodb-user
The segfault on one of the machines:

Aug 7 23:40:54 lo24-sv-13 kernel: mongos[30474]: segfault at
0000000000368d58 rip 00000034a52711d8 rsp 0000000067049d00 error 4

I configured each app on an appserver to fetch 75 connections from the
mongos on localhost. On the two mongod shards (which are single node
replica sets) mongostat prints out around 13000 connections each,
which is really a lot since 80*75 should be far below 8000.

For us it seem mongos is really unstable.

breadfan

unread,
Aug 9, 2010, 5:32:13 AM8/9/10
to mongodb-user
On Saturday (high traffic) we had a lot of too many open files for
each of the servers running the mongos. The client app is configured
to use only 75 connection to the mongos on the same machine. Wondering
why this is a problem, cause then shards are running on dedicated,
different machines.

We raised the count from 1024 to 30000 but the segfaults are still
present.

Eliot Horowitz

unread,
Aug 9, 2010, 9:10:42 AM8/9/10
to mongod...@googlegroups.com
Is there more of a stack trace or is that?

Can you try this build:
http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debugsymbols-1.6.0.tgz
Has more debugging symbols

breadfan

unread,
Aug 9, 2010, 10:17:57 AM8/9/10
to mongodb-user
That's everything. We'll try the debug version and post the results.
Thanks for the fast response!

breadfan

unread,
Aug 11, 2010, 4:02:40 AM8/11/10
to mongodb-user
Created a bug for this. more available there: http://jira.mongodb.org/browse/SERVER-1593
Reply all
Reply to author
Forward
0 new messages