Shard balancing stalled - no locks - no log errors?
6 views
Skip to first unread message
James Peter
Oct 12, 2011, 9:39:06 AM10/12/11
to mongodb-user
Hmmm... I'm a bit stumped on this one.
I've got two collections that were across 23 shards. I just added 20
new shards as I was running out of space. One of the collections has
rebalanced normally, but the other appears to have stalled. One shard
has 15 chunks, some have 14 other have 8 and 9.
Sharding status;
http://pastebin.com/PusmJCuU
In cool graph form:
http://i.imgur.com/9ryVx.png
tc.twitter_cache is as expected, co.connections is clearly not. It's
been stuck like this for a good 24 hours after about 20 hours of
normal rebalancing.
There's no stale locks in the locks table (I had this problem once
before and cleaned out the locks to get the process started again, but
none in there this time). Only the balancer;
Array
(
[_id] => balancer
[process] => mffront6:28349:1318391291:1804289383
[state] => 0
[ts] => MongoId Object
(
[$id] => 4e9597169feb93344edceaef
)
[when] => MongoDate Object
(
[sec] => 1318426390
[usec] => 32000
)
[who] => mffront6:28349:1318391291:1804289383:Balancer:846930886
[why] => doing balance round
)
I can't find anything unusual in the log files anywhere but with 70
odd distributed log files it's tough to figure out where to look. Is
there anything I can grep for?
Cheers,
James
I've got two collections that were across 23 shards. I just added 20
new shards as I was running out of space. One of the collections has
rebalanced normally, but the other appears to have stalled. One shard
has 15 chunks, some have 14 other have 8 and 9.
Sharding status;
http://pastebin.com/PusmJCuU
In cool graph form:
http://i.imgur.com/9ryVx.png
tc.twitter_cache is as expected, co.connections is clearly not. It's
been stuck like this for a good 24 hours after about 20 hours of
normal rebalancing.
There's no stale locks in the locks table (I had this problem once
before and cleaned out the locks to get the process started again, but
none in there this time). Only the balancer;
Array
(
[_id] => balancer
[process] => mffront6:28349:1318391291:1804289383
[state] => 0
[ts] => MongoId Object
(
[$id] => 4e9597169feb93344edceaef
)
[when] => MongoDate Object
(
[sec] => 1318426390
[usec] => 32000
)
[who] => mffront6:28349:1318391291:1804289383:Balancer:846930886
[why] => doing balance round
)
I can't find anything unusual in the log files anywhere but with 70
odd distributed log files it's tough to figure out where to look. Is
there anything I can grep for?
Cheers,
James
James Peter
Oct 12, 2011, 9:42:15 AM10/12/11
to mongodb-user
I'm running 2.0.0 BTW. I haven't set a size on any of the shards.
Eliot Horowitz
Oct 12, 2011, 10:32:05 AM10/12/11
to mongod...@googlegroups.com
THe balancer won't be aggressive until there is an 8 chunk differential.
If you want data to be smoother, you can lower the chunk size.
If you do - I would run a bunch of manual splits to get everything to
the new size.
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
James Peter
Oct 12, 2011, 8:38:08 PM10/12/11
to mongodb-user
Ah cool - missed that in the docs.
Out of interest - does the balancer migrate and split multiple chunks
at once or is it always one at a time?
Out of interest - does the balancer migrate and split multiple chunks
at once or is it always one at a time?
Kristina Chodorow
Oct 13, 2011, 2:25:27 PM10/13/11
to mongodb-user
Multiple splits can happen at the same time, but it'll only migrate
one at a time.
one at a time.
Reply all
Reply to author
Forward
0 new messages