mongodump issue in shard environement (process almost fail / is terribly slow)

[Ludovic Fleury]

Hey,

We need some help here for a "special" operation.:
We're moving from 1 mongos + 20 single shards to 3 mongos + 3 replicated shards.

We're using mongodump from the old mongos to grab 800 gigs of data (one collection with a native ObjectId as _id).

• The balancer is disabled
• The dump is "over the wire", we dump from the mongos to another server

Yet the process "almost" fail or is terribly slow.

While monitoring the process, we find out that only one shard seems "active" (ie: network activity and i/o activity, others shard are "sleeping").

iostat on shard03 : 110r/s , 1w/s
iostat on shard04 : 3r/s , 1w/s
...
iostat on shard08 : 2r/s , 2w/s
...
iostat on shard14 : 1r/s , 1w/s
...
iostat on shard20 : 1r/s , 1w/s

On the shard 3 which seems "active":

• Proc usage is low: sys 2%, user 1%, idle 126%, wait 72%
  
• I/O are low: 110 reads/sec, 1 writes/sec
• ram usage is low
• load average is also low
• mms average: 80 in "page faults" and 4Mbits in "network"
  

On the server dumping (receiving the data):

• Proc usage is low.
  
• I/O are low: 0r/s , 6w/s
• ram usage is low
  

[Ludovic Fleury]

Any idea ? we have no clue :(

[Siddharth Singh]

Given your use case, there is no other straightforward way to do it except using mongodump like you are using.

Since mongodump has to dump 800 gigs of data it is also expected that it will take a while to finish.

mongodump dumps one collection at a time by opening a cursor on mongos. Depending on if

the collection is sharded or not this may involve mongos talking to one or more shards and so you 

might see one or more shards active at any point of time.