I've just upgraded from 1.6.5
Everything looks like going fine, (and much better ).
I had problems with shard chunk copy and splitting (very low
performance). I readed that with 1.8.0 all this will go much better so
I upgraded.
The problem now is that looks like something wrong is going on with
spliting:
At some point on one of my partitions appears this message:
Tue Mar 15 10:45:15 [conn568] request split points lookup for chunk
crawler.ad { : 10662019385342 } -->> { : MaxKey }
And at the same time, on the two mongos processes being used start to
appear lots of this messages:
....
/home/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x5e6) [0x6680b6]
/home/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x141) [0x67ae11]
/home/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x272) [0x5817a2]
/home/mongo/bin/mongos(thread_proxy+0x80) [0x6a26f0]
/lib64/libpthread.so.0 [0x386c406367]
/lib64/libc.so.6(clone+0x6d) [0x386bcd309d]
Tue Mar 15 10:45:56 [conn14] AssertionException in process:
setShardVersion failed! { "errmsg" : "not master", "ok" : 0 }
Tue Mar 15 10:45:57 [conn14] update failed b/c of
StaleConfigException, retrying left:4 ns:
crawler.ad query: { _id:
662005864631 }
Tue Mar 15 10:45:57 [conn14] ns:
crawler.ad ClusteredCursor::query
ShardConnection had to change attempt: 0
Tue Mar 15 10:45:58 [conn17] ns:
crawler.ad ClusteredCursor::query
ShardConnection had to change attempt: 0
Tue Mar 15 10:45:58 [conn14] setShardVersion failed: { errmsg:
"not master", ok: 0.0 }
Tue Mar 15 10:45:58 [conn14] Assertion: 10429:setShardVersion failed!
{ "errmsg" : "not master", "ok" : 0 }
0x522c19 0x69ee23 0x69e90b 0x69e90b 0x69e90b 0x5799e6 0x57a2f6
0x57781d 0x6680b6 0x67ae11 0x5817a2 0x6a26f0 0x386c406367
0x386bcd309d
/home/mongo/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x522c19]
/home/mongo/bin/mongos [0x69ee23]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/
mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi
+0x16) [0x5799e6]
/home/mongo/bin/
mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)
[0x57a2f6]
/home/mongo/bin/
mongos(_ZN5mongo15ShardConnection25checkMyConnectionVersionsERKSs
+0x3d) [0x57781d]
/home/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x5e6) [0x6680b6]
/home/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x141) [0x67ae11]
/home/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x272) [0x5817a2]
/home/mongo/bin/mongos(thread_proxy+0x80) [0x6a26f0]
/lib64/libpthread.so.0 [0x386c406367]
/lib64/libc.so.6(clone+0x6d) [0x386bcd309d]
Tue Mar 15 10:45:58 [conn14] AssertionException in process:
setShardVersion failed! { "errmsg" : "not master", "ok" : 0 }
Tue Mar 15 10:45:58 [conn17] setShardVersion failed: { errmsg:
"not master", ok: 0.0 }
Tue Mar 15 10:45:58 [conn17] Assertion: 10429:setShardVersion failed!
{ "errmsg" : "not master", "ok" : 0 }
0x522c19 0x69ee23 0x69e90b 0x69e90b 0x69e90b 0x5799e6 0x57a2f6
0x57781d 0x6680b6 0x67ae11 0x5817a2 0x6a26f0 0x386c406367
0x386bcd309d
/home/mongo/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x522c19]
/home/mongo/bin/mongos [0x69ee23]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/
mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi
+0x16) [0x5799e6]
/home/mongo/bin/
mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)
[0x57a2f6]
/home/mongo/bin/
mongos(_ZN5mongo15ShardConnection25checkMyConnectionVersionsERKSs
+0x3d) [0x57781d]
/home/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x5e6) [0x6680b6]
/home/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x141) [0x67ae11]
/home/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x272) [0x5817a2]
/home/mongo/bin/mongos(thread_proxy+0x80) [0x6a26f0]
/lib64/libpthread.so.0 [0x386c406367]
/lib64/libc.so.6(clone+0x6d) [0x386bcd309d]
Tue Mar 15 10:45:58 [conn17] AssertionException in process:
setShardVersion failed! { "errmsg" : "not master", "ok" : 0 }
Tue Mar 15 10:45:58 [conn14] ns:
crawler.ad ClusteredCursor::query
ShardConnection had to change attempt: 0
Tue Mar 15 10:45:58 [conn14] setShardVersion failed: { errmsg:
"not master", ok: 0.0 }
Tue Mar 15 10:45:58 [conn14] Assertion: 10429:setShardVersion failed!
{ "errmsg" : "not master", "ok" : 0 }
0x522c19 0x69ee23 0x69e90b 0x69e90b 0x69e90b 0x5799e6 0x57a2f6
0x57781d 0x6680b6 0x67ae11 0x5817a2 0x6a26f0 0x386c406367
0x386bcd309d
/home/mongo/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x522c19]
/home/mongo/bin/mongos [0x69ee23]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/
mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi
+0x16) [0x5799e6]
/home/mongo/bin/
mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)
[0x57a2f6]
/home/mongo/bin/
mongos(_ZN5mongo15ShardConnection25checkMyConnectionVersionsERKSs
+0x3d) [0x57781d]
/home/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x5e6) [0x6680b6]
/home/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x141) [0x67ae11]
/home/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x272) [0x5817a2]
/home/mongo/bin/mongos(thread_proxy+0x80) [0x6a26f0]
/lib64/libpthread.so.0 [0x386c406367]
/lib64/libc.so.6(clone+0x6d) [0x386bcd309d]
Tue Mar 15 10:45:58 [conn14] AssertionException in process:
setShardVersion failed! { "errmsg" : "not master", "ok" : 0 }
Tue Mar 15 10:45:59 [LockPinger] dist_lock pinged successfully for:
OBE010.local:1300111198:1804289383
Tue Mar 15 10:45:59 [conn14] ns:
crawler.ad ClusteredCursor::query
ShardConnection had to change attempt: 0
Tue Mar 15 10:45:59 [conn14] setShardVersion failed: { errmsg:
"not master", ok: 0.0 }
Tue Mar 15 10:45:59 [conn14] Assertion: 10429:setShardVersion failed!
{ "errmsg" : "not master", "ok" : 0 }
0x522c19 0x69ee23 0x69e90b 0x69e90b 0x69e90b 0x5799e6 0x57a2f6
0x57781d 0x6680b6 0x67ae11 0x5817a2 0x6a26f0 0x386c406367
0x386bcd309d
/home/mongo/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x522c19]
/home/mongo/bin/mongos [0x69ee23]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/mongos [0x69e90b]
/home/mongo/bin/
mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi
+0x16) [0x5799e6]
/home/mongo/bin/
mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)
[0x57a2f6]
/home/mongo/bin/
mongos(_ZN5mongo15ShardConnection25checkMyConnectionVersionsERKSs
+0x3d) [0x57781d]
/home/mongo/bin/mongos(_ZN5mongo7Request7processEi+0x5e6) [0x6680b6]
/home/mongo/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x141) [0x67ae11]
/home/mongo/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x272) [0x5817a2]
/home/mongo/bin/mongos(thread_proxy+0x80) [0x6a26f0]
/lib64/libpthread.so.0 [0x386c406367]
/lib64/libc.so.6(clone+0x6d) [0x386bcd309d]
Tue Mar 15 10:45:59 [conn14] AssertionException in process:
setShardVersion failed! { "errmsg" : "not master", "ok" : 0 }
Tue Mar 15 10:45:59 [conn13] AssertionException in process: ns:
crawler.ad ClusteredCursor::query ShardConnection had to change
Tue Mar 15 10:46:01 [conn14] AssertionException in process: ns:
crawler.ad ClusteredCursor::query ShardConnection had to change
At this point, stops sending messages until after some time another:
request split points lookup for chunk
crawler.ad { : 10662019385342 }
-->> { : MaxKey }
Appears and everything repeats again.
It's doing this every 5 minutes or so...
* Is the splitting working?
* If not, there's any way to solve this?