Data per Chunk Exceeded Maximum Chunk Size

[Nisrina Luthfiyati]

Hi all,

I'm testing mongoDB sharding with 2 shards and inserted around 1,000,000 documents.
My maximum chunk size is set at default, 64 MB, however, I found that the data size per chunk is way bigger than that.

Here's the sharding distribution:

--- coba.coba.pageView

Shard replA at replA/replA1.com:27017,replA2.com:27017
 data : 1.94GiB docs : 539261 chunks : 17
 estimated data per chunk : 117.02MiB
 estimated docs per chunk : 31721

Shard replB at replB/replB1.com:27017,replB2.com:27017
 data : 1.69GiB docs : 470715 chunks : 11
 estimated data per chunk : 157.87MiB
 estimated docs per chunk : 42792

Totals
 data : 3.63GiB docs : 1009976 chunks : 28
 Shard replA contains 53.39% data, 53.39% docs in cluster, avg obj size on shard : 3KiB
 Shard replB contains 46.6% data, 46.6% docs in cluster, avg obj size on shard : 3KiB

It looked like the chunk would not split even when the size exceeded the maximum size.
The shard key is hashed, so I'm quite sure it's not a key granularity issue.

Any idea why? Would I need to split chunks manually?

Any help would be greatly appreciated, thank you!

Nisrina

[thanzeer mohammed]

Couple of reason I can think of are

1) split threshold never reached due to mongos going up and down very frequently.
2)  All 3 config servers are not healthy.

[Nisrina Luthfiyati]

Thanks for the reply Thanzeer.

Anyway I think I found out why.
After checking the log (mongodb.log) I found a lot of splitChunk failed warning:

Thu Oct 16 12:27:20.230 [conn1264220] warning: splitChunk failed - cmd: { splitChunk: "monitor.coba.coba.pageView", keyPattern: { _id: "hashed" }, min: { _id: 7171816218909932292 }, max: { _id: 7317718316615131599 }, from: "replB", splitKeys: [ { _id: 7246347097818303012 } ], shardId: "monitor.coba.coba.pageView-_id_7171816218909932292", configdb: "conftest01.com:27019" } result: { who: { _id: "monitor.coba.coba.pageView", process: "replA1.com:27017:1411612695:469181041", state: 2, ts: ObjectId('543fb9a8ddb16c7c92e658ed'), when: new Date(1413462440122), who: "replA1.com:27017:1411612695:469181041:conn74596:985601791", why: "migrate-{ _id: -8258558921988299825 }" }, ok: 0.0, errmsg: "the collection's metadata lock is taken" }

Apparently it's caused by some stale locks. I tried removing the balancer lock:

> use config
> db.locks.remove({_id:"balancer"})

, reinserted the documents and the data per chunk looked to be under 64 MB.

This thread helps if any of you happen to run into the same problem: https://groups.google.com/forum/#!topic/mongodb-user/yTH49XVrsjM

[Asya Kamsky]

What made you think the locks were stale? It looks like there was a
migration in progress, and it's definitely NOT a good idea to remove
locks which are there to prevent access to data that's exclusively
locked for a particular active process.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/96146c95-f65f-4be6-b561-7672b75b87fa%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Nisrina Luthfiyati]

Asya,
I run sh.isBalancerRunning() and it returned false.
Left it overnight and did not see any chunk movement across shards or any chunk splitting.
Now that I think about it I should also check the timestamp or just turn off the balancer all the way.
Would this be enough to determine whether the lock was stale or not?
Thank you.

[Asya Kamsky]

The timestamps are certainly telling but you would need to check
mongod logs as well to see if there is any migration related activity.

In config db the changelog collection also gets written to when
migrations start, and end (whether successfully or aborted).

Asya

> https://groups.google.com/d/msgid/mongodb-user/1528cad0-1c01-4121-939f-9dfb3a63f1c0%40googlegroups.com.