Troubles with some high latency since i have added a new shard

[Rhaven]

Hello guys,

I'm stuck with some high latency since i have added a new shard(rs3) to my cluster, specially when the balancer is up and may running some moveChunk command every time he need to restore the balance of data between each shard. i tried to define a balance window but nothing changed. In my log, the balance window was setup from 06:00 am to 09:00 am.

Actually the balancer is set to false while i try to find out what is going on..

My cluster looks like this:

server 1 : 1 mongod (part of rs1), 1 mongod configsvr, 1 mongod arbiter(for rs2)

server 2 : 1 mongod (part of rs1), 1 mongod configsvr, 1 mongos

server 3 : 1 mongod (part of rs2), 1 mongod configsvr, 1 mongod arbiter(for rs1)

server 4 : 1 mongod (part of rs2), 1 mongos

//recently added

server 5 : 1 mongod (part of rs3)

server 6 : 1 mongod (part of rs3), 1 mongod arbiter (for rs3)

I'm using mongodb in 2.2.0 version. 

The chunksize is the default one (64mb). 

An idea of the balance on a sharded collection :

mydb.col_a chunks:

rs3 4

rs2 547

rs1 541

Log from mongos on server4 there : http://pastebin.com/6GN7vtfw

Log from mongod(rs1) on server2 there : http://pastebin.com/qaWLjza7

Log from mongod(rs1) on server1 there : http://pastebin.com/DbQ2mtXn

[David Hows]

Hi Rhaven,

What do you mean by "high latency" how are you measuring this and between what points?

Your cluster looks terribly unbalanced, which is why the balancer would be working hard when its on. Can you post your balancer configuration so we can see how you have your timing configuration setup.

Do you have MMS, can you post your group name or your MMS url?

Have you considered manually balancing some of the shards yourself using the moveChunk command, this way you can control the load of balancing until your cluster is more stable, then re-enable the balancer. Documentation on moveChunk is available http://docs.mongodb.org/manual/reference/command/moveChunk

Hope that helps

David