Huge collection is not balanced across shards

[sunny]

Hello,

We have several collections on 40 shards. All the collections but one
are balanced across the shards.
However, collection is not balanced, even though it already contains
7GB of data in one chunk. As you can see, another collection is
balanced just fine.

What's wrong? We're using Mongo 1.6.5 and have 2 mongos and 3 mongo-
config servers.

Also, what the "t" and "i" in printShardingStatus() mean?

-------------------------------------------

> db.addressbooks.stats()
{
"sharded" : true,
"ns" : "db.addressbooks",
"count" : 253560,
"size" : 7110284088,
"avgObjSize" : 28041.820823473736,
"storageSize" : 7253458688,
"nindexes" : 1,
"nchunks" : 1,
"shards" : {
"set1" : {
"ns" : "db.addressbooks",
"count" : 253560,
"size" : 7110284088,
"avgObjSize" : 28041.820823473736,
"storageSize" : 7253458688,
"numExtents" : 31,
"nindexes" : 1,
"lastExtentSize" : 1217185536,
"paddingFactor" : 1.5699999999985885,
"flags" : 1,
"totalIndexSize" : 25862144,
"indexSizes" : {
"_id_" : 25862144
},
"ok" : 1
}
},
"ok" : 1
}


> db.printShardingStatus()
--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{ "_id" : "set1", "host" : "set1/
amdbm001:10000,amdbm002:10000" }
{ "_id" : "set2", "host" : "set2/
amdbm003:10000,amdbm004:10000" }
{ "_id" : "set3", "host" : "set3/
amdbm005:10000,amdbm006:10000" }
{ "_id" : "set4", "host" : "set4/
amdbm007:10000,amdbm008:10000" }
{ "_id" : "set5", "host" : "set5/
amdbm009:10000,amdbm010:10000" }
[...]
{ "_id" : "set37", "host" : "set37/
amdbm073:10000,amdbm074:10000" }
{ "_id" : "set38", "host" : "set38/
amdbm075:10000,amdbm076:10000" }
{ "_id" : "set39", "host" : "set39/
amdbm077:10000,amdbm078:10000" }
{ "_id" : "set40", "host" : "set40/
amdbm079:10000,amdbm080:10000" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" :
"config" }
{ "_id" : "db", "partitioned" : true, "primary" : "set1" }
db.addressbooks chunks:
{ "_id" : { $minKey : 1 } } -->> { "_id" :
{ $maxKey : 1 } } on : set1 { "t" : 1000, "i" : 0 }
[...]
db.users chunks:
{ "_id" : { $minKey : 1 } } -->> { "_id" :
"0000262307b1a245e73e759e799e55c20451f031" } on : set10 { "t" : 2000,
"i" : 0 }
{ "_id" :
"0000262307b1a245e73e759e799e55c20451f031" } -->> { "_id" :
"0ffb0789d68efe3e56c5a8b4f4ec8bf1303c5f9c" } on : set20 { "t" : 13000,
"i" : 0 }
{ "_id" :
"0ffb0789d68efe3e56c5a8b4f4ec8bf1303c5f9c" } -->> { "_id" :
"200781b0839e69369198c65bc259a38f97d67f66" } on : set12 { "t" : 13000,
"i" : 1 }
{ "_id" :
"200781b0839e69369198c65bc259a38f97d67f66" } -->> { "_id" :
"2ffbc8d2d54b1a1bbdc23e77a1a064e8675b35b2" } on : set21 { "t" : 14000,
"i" : 0 }
{ "_id" :
"2ffbc8d2d54b1a1bbdc23e77a1a064e8675b35b2" } -->> { "_id" :
"3ff6625265e74246ecd91313e9b6aaf806b85b20" } on : set13 { "t" : 14000,
"i" : 1 }
{ "_id" :
"3ff6625265e74246ecd91313e9b6aaf806b85b20" } -->> { "_id" :
"4ff2698377e5cb38c16a58a37a684ba6401cd7d8" } on : set19 { "t" : 11000,
"i" : 0 }
{ "_id" :
"4ff2698377e5cb38c16a58a37a684ba6401cd7d8" } -->> { "_id" :
"6001792911030bf58de6728ff9c6564d662e0c0e" } on : set14 { "t" : 11000,
"i" : 1 }
{ "_id" :
"6001792911030bf58de6728ff9c6564d662e0c0e" } -->> { "_id" :
"6ff51e724db7aef995efd0c2b8a0fe333efbbc89" } on : set24 { "t" : 17000,
"i" : 0 }
{ "_id" :
"6ff51e724db7aef995efd0c2b8a0fe333efbbc89" } -->> { "_id" :
"7ff33caf1d4e46de0afd49fda82e5e520ed00c9f" } on : set15 { "t" : 17000,
"i" : 1 }
{ "_id" :
"7ff33caf1d4e46de0afd49fda82e5e520ed00c9f" } -->> { "_id" :
"8fedc025aed6e4ad7c54d77bb8ff04be54b74946" } on : set22 { "t" : 15000,
"i" : 0 }
{ "_id" :
"8fedc025aed6e4ad7c54d77bb8ff04be54b74946" } -->> { "_id" :
"9ff013fc63f30d47bfc6184da10c96eadbc69810" } on : set16 { "t" : 15000,
"i" : 1 }
{ "_id" :
"9ff013fc63f30d47bfc6184da10c96eadbc69810" } -->> { "_id" :
"aff313b70ab8b5efa0355cfcd27487d6800a56e5" } on : set2 { "t" : 12000,
"i" : 0 }
{ "_id" :
"aff313b70ab8b5efa0355cfcd27487d6800a56e5" } -->> { "_id" :
"b80805c65da45d46f73f34ded972afb74bbacbff" } on : set26 { "t" : 19000,
"i" : 0 }
{ "_id" :
"b80805c65da45d46f73f34ded972afb74bbacbff" } -->> { "_id" :
"c015e62f5faad2ce462274df2ce0df56955a2dfe" } on : set17 { "t" : 19000,
"i" : 1 }
{ "_id" :
"c015e62f5faad2ce462274df2ce0df56955a2dfe" } -->> { "_id" :
"d01653959346e5488f4e5369bfead4448be9dbac" } on : set25 { "t" : 18000,
"i" : 0 }
{ "_id" :
"d01653959346e5488f4e5369bfead4448be9dbac" } -->> { "_id" :
"e00ffb2e6541107dda21f92a9db65c51c683b2ef" } on : set18 { "t" : 18000,
"i" : 1 }
{ "_id" :
"e00ffb2e6541107dda21f92a9db65c51c683b2ef" } -->> { "_id" :
"f0119a3045a9df755c15f519665f226a74e5ad11" } on : set23 { "t" : 16000,
"i" : 0 }
{ "_id" :
"f0119a3045a9df755c15f519665f226a74e5ad11" } -->> { "_id" :
"ffffc72cb39a4790983e7b812fdaff55a8b46788" } on : set1 { "t" : 16000,
"i" : 1 }
{ "_id" :
"ffffc72cb39a4790983e7b812fdaff55a8b46788" } -->> { "_id" :
{ $maxKey : 1 } } on : set11 { "t" : 3000, "i" : 0 }

[Nat]

You should check mongod and mongos log files to see whether there are
any error/warning messages.

[Eliot Horowitz]

There may have been a splitting error.
You may want to try 1.7.6 as it addresses some of the cases where a
collection doesn't get split properly.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Eliot Horowitz]

t and i are part of the timestamp, t is time, i is an incremented counter
For sharding, its major and minor timestamp, major for migrates, minor
for splits.

On Wed, Feb 23, 2011 at 4:39 AM, sunny <sun...@gmail.com> wrote:

[sunny]

Should we upgrade mongos or also mongod?
1.7.6 is "unstable". Is it close to being production-ready?
Should we split the huge collection manually?

> > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Eliot Horowitz]

If you upgraded, you would have to both.
It is very close to being stable (1.8.0)
Spliting manually is another valid option.

[sunny]

Thanks
We'll try to update to 1.7.6.
Once we do, mongo will split the huge chunk?
We've attempted to split chunk manually, only to get 2 chunks - one of
which is empty (we used the "find" alternative)

> >> > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.-Hide quoted text -

[Eliot Horowitz]

What is the shard key?
You'll have to keep inserting/updating for it to get split.
Can you try manual splitting with middle?

[sunny]

The shard key is _id (160 bit SHA1 unique ID).
There would be more inserts to this collection.
Do we need to do the manual split at all if we upgrade?

> >> >> > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.-Hidequoted text -

[Eliot Horowitz]

If you keep inserting, mongo should handle splitting on its own.
Hard to guarantee without knowing why its not working right now, but probably.

[sunny]

We've manually split the huge chunk and mongo finally successfully re-
balanced.
It also seems to be splitting chunks on its own now.

Thanks

> ...
>
> read more »