Hi.
I try to understand why my mongodb config server produce hight IO on disk. My question seems like
Very high disk IO on Mongodb Config servers topic.
typical iostat -xnt 3 output:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0,00 11,67 0,00 205,00 0,00 0,24 2,39 1,21 3,51 0,00 3,51 2,58 52,80
This is a test environment and i have no troubles until i do some load tests.
I make a few experiments and definetly sure that:
- my config server disk load independent from work load
- when i stop balancer disk load from my config server like 0, then i start balancer and disk load high again
- when balancer enebled his status is blinked 0/1/2/0/1/2 all time
I have cluster with 6 shards and only one sharded database - apikeys with 99 sharded collections, all collections are even:
sh.status()
apikeys.hourly_counters99
shard key: { "s" : 1 }
chunks:
ps-rs2 2
ps-rs3 2
ps-rs4 1
ps-rs5 1
ps-rs6 1
ps-rs1 2
{ "s" : { "$minKey" : 1 } } -->> { "s" : 2 } on : ps-rs2 Timestamp(2, 0)
{ "s" : 2 } -->> { "s" : 3 } on : ps-rs3 Timestamp(3, 0)
{ "s" : 3 } -->> { "s" : 4 } on : ps-rs4 Timestamp(4, 0)
{ "s" : 4 } -->> { "s" : 5 } on : ps-rs5 Timestamp(5, 0)
{ "s" : 5 } -->> { "s" : 6 } on : ps-rs6 Timestamp(6, 0)
{ "s" : 6 } -->> { "s" : 7 } on : ps-rs2 Timestamp(7, 0)
{ "s" : 7 } -->> { "s" : 8 } on : ps-rs3 Timestamp(8, 0)
{ "s" : 8 } -->> { "s" : 9 } on : ps-rs1 Timestamp(8, 1)
{ "s" : 9 } -->> { "s" : { "$maxKey" : 1 } } on : ps-rs1 Timestamp(1, 16)
Every hour i write data to a new collection and do some statistic count. Data look like this
db.hourly_counters1.find()
{ "_id" : 1, "c" : 1, "s" : 1, "v" : 0 }
{ "_id" : 2, "c" : 1, "s" : 2, "v" : 0 }
{ "_id" : 3, "c" : 1, "s" : 3, "v" : 0 }
{ "_id" : 4, "c" : 1, "s" : 4, "v" : 0 }
{ "_id" : 5, "c" : 1, "s" : 5, "v" : 0 }
{ "_id" : 12, "c" : 1, "s" : 12, "v" : 0 }
{ "_id" : 6, "c" : 1, "s" : 6, "v" : 0 }
{ "_id" : 7, "c" : 1, "s" : 7, "v" : 0 }
{ "_id" : 21, "c" : 2, "s" : 3, "v" : 0 }
{ "_id" : 22, "c" : 2, "s" : 4, "v" : 0 }
s variable can be 0 - 18 (i know that's not perfect sharded).
I have a log file with systemlog verbosity: 5. Default verbosity show only fsync message and not informative.
I have mongodb 2.6.11 and i try 2.6.4, 2.6.5 versions. I completly recreate cluster and restore data.
If u have any questions, welcome :)
And thx for help.