mongodb config server produse hight IO on disk

[Alexander]

Hi.

I try to understand why my mongodb config server produce hight IO on disk. My question seems like Very high disk IO on Mongodb Config servers topic.

typical iostat -xnt 3 output:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0,00    11,67    0,00  205,00     0,00     0,24     2,39     1,21    3,51    0,00    3,51   2,58  52,80

This is a test environment and i have no troubles until i do some load tests.

I make a few experiments and definetly sure that:
- my config server disk load independent from work load
- when i stop balancer disk load from my config server like 0, then i start balancer and disk load high again
- when balancer enebled his status is blinked 0/1/2/0/1/2 all time

I have cluster with 6 shards and only one sharded database - apikeys with 99 sharded collections, all collections are even:
sh.status()
apikeys.hourly_counters99
shard key: { "s" : 1 }
               chunks:
                ps-rs2    2
                ps-rs3    2
                ps-rs4    1
                ps-rs5    1
                ps-rs6    1
                ps-rs1    2
            { "s" : { "$minKey" : 1 } } -->> { "s" : 2 } on : ps-rs2 Timestamp(2, 0)
            { "s" : 2 } -->> { "s" : 3 } on : ps-rs3 Timestamp(3, 0)
            { "s" : 3 } -->> { "s" : 4 } on : ps-rs4 Timestamp(4, 0)
            { "s" : 4 } -->> { "s" : 5 } on : ps-rs5 Timestamp(5, 0)
            { "s" : 5 } -->> { "s" : 6 } on : ps-rs6 Timestamp(6, 0)
            { "s" : 6 } -->> { "s" : 7 } on : ps-rs2 Timestamp(7, 0)
            { "s" : 7 } -->> { "s" : 8 } on : ps-rs3 Timestamp(8, 0)
            { "s" : 8 } -->> { "s" : 9 } on : ps-rs1 Timestamp(8, 1)
            { "s" : 9 } -->> { "s" : { "$maxKey" : 1 } } on : ps-rs1 Timestamp(1, 16)

Every hour i write data to a new collection and do some statistic count. Data look like this
db.hourly_counters1.find()
{ "_id" : 1, "c" : 1, "s" : 1, "v" : 0 }
{ "_id" : 2, "c" : 1, "s" : 2, "v" : 0 }
{ "_id" : 3, "c" : 1, "s" : 3, "v" : 0 }
{ "_id" : 4, "c" : 1, "s" : 4, "v" : 0 }
{ "_id" : 5, "c" : 1, "s" : 5, "v" : 0 }
{ "_id" : 12, "c" : 1, "s" : 12, "v" : 0 }
{ "_id" : 6, "c" : 1, "s" : 6, "v" : 0 }
{ "_id" : 7, "c" : 1, "s" : 7, "v" : 0 }
{ "_id" : 21, "c" : 2, "s" : 3, "v" : 0 }
{ "_id" : 22, "c" : 2, "s" : 4, "v" : 0 }

s variable can be 0 - 18 (i know that's not perfect sharded).

I have a log file with systemlog verbosity: 5. Default verbosity show only fsync message and not informative.
I have mongodb 2.6.11 and i try 2.6.4, 2.6.5 versions. I completly recreate cluster and restore data.

If u have any questions, welcome :)
And thx for help.

[Asya Kamsky]

> when i stop balancer disk load from my config server like 0, then i start balancer and disk load high again

Don't you think that maybe this is normal?  When the balancer is running, it's logging various information about what it's doing to the config server (among other things).

When the balancer is not running, there is no disk load...

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/11fcc8a1-82b7-4fca-99f7-c21d25a4c4b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander]

Don't you think that maybe this is normal?  When the balancer is running, it's logging various information about what it's doing to the config server (among other things).

I think this is normal when i have work load or when i whait some time after stop work load. Maby it is not normal when i create new cluster, shard database, presplit collections, mongorestore data and not give work load for a long time.

So i am a worried about it and try to understand why this happens, maby i do smth wrong.

понедельник, 14 сентября 2015 г., 18:22:47 UTC+3 пользователь Asya Kamsky написал:

[Asya Kamsky]

You didn't mention 

> presplit collections

You said the balancer was running (stopping the balancer stops the load) that means that data was NOT evenly distributed when the load finished and the balancer is moving data to achieve equal amount of data on each shard.

Can you explain how you are presplitting and whether the data appears balanced when the restore finishes?

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6263e0b1-552a-447c-8089-2d82e1ece915%40googlegroups.com.

[Alexander]

After I restore the data and wait for a long time I see the picture as I described above. Disable the balancer removes the disk load. If the balancer enebled, its state changes very often - blinked. db.locks.find( { _id : "balancer" } ).pretty() show me state often: "state" : 0, then 1, then 2.

1. I create cluster with 6 shards, 3 config servers, 3 mongos and have only db admin.
2. Disable balancer by sh.stopBalancer()
3. Create database and shard it.
sh.enableSharding("apikeys")
for ( var x=1; x<100; x++ ) {
    var name = "hourly_counters" + x ;
    var dbcoll = "apikeys" + "." + name ;
    sh.shardCollection( dbcoll, { "s" : 1 } ) ;     for (var y=2; y<10; y++) {
        db.runCommand( { split : dbcoll, middle : { s : y } } );    
    }
}
Now i have all 9 chunks on one shard.
4. maby here is not the same script but i use idea that i have primery shard and move chunks to other except last.
var shServer = [ "ps-rs5", "ps-rs2", "ps-rs3", "ps-rs4", "ps-rs6"]

for ( var x=1; x<100; x++ ) {
   var name = "hourly_counters" + x ;
   var dbcoll = "apikeys" + "." + name;
   for (var y=1; y<9; y++) {
       var shard = y%5
       db.adminCommand({moveChunk : dbcoll, find :  { s : y }, to : shServer[shard]})
   }
}

5. eneble balancer sh.startBalancer()
6. mongorestore data without --drop option

вторник, 15 сентября 2015 г., 19:27:33 UTC+3 пользователь Asya Kamsky написал: