Mongorestore 3.0.2 blocks other operations

[Maziyar PANAHI]

Hello all,

I migrated my replicaset (2 Mongod + arbiter) to mongodb 3.0.2 with wiredTiger storage engine with default config 2 weeks ago and I was trying to restore some of my big collections since I have more space now. Primary is on SSD RAID (3TB/ used only 600MB used by now) and the secondary is on a big RAID over fast disk 15K with the same size.

I started my mongorestore (collection size is about 400 million) and after 14 hours I noticed all the operations were stopped. I guess they have been queued for sometime and every minute or so you see bunch of them on mongostat. I even restarted all my applications (Node.js latest driver) but no luck. No updates, no inserts and not query. These operations are really small. Like 10 queries/s or 30 update/s. 

This is scary cause I was down for the next 14 to 15 hours till the restore was completed and I have two other collections with the same size.

Here are some screenshots of the operations and how I ran the mongorestore:

Secondary doing inserts:

And Primary doing nothing or maybe something at the background:

and once in while it will insert like this:

and this is how it looks like when all the operations backed up and blocked by mogorestore:

And some MMS:

And finally this a command I ran mongostore and how I started my replicaSet:

mongorestore --host 10.0.0.4 --collection FRTweets_2014 --db test --numParallelCollections 10 --numInsertionWorkersPerCollection 10 --writeConcern 0 /dump/test/FRTweets.bson

sudo mongod --storageEngine wiredTiger --dbpath /data/ --replSet rs0 --fork --logpath /var/log/mongodb/fork.log

Is this a bug in new Mongorestore or I am doing something wrong that blocks other operations?

Thanks

[Asya Kamsky]

Hi,

If I understand correctly, you started mongorestore and you overrode its defaults to make it send work at the server as fast as it possibly could:

> --numParallelCollections 10 

> --numInsertionWorkersPerCollection 10 

> --writeConcern 0

This says restore 10 collections in parallel, use 10 workers (threads) per collection and writeConcern 0 says do NOT wait for the inserts to keep up, instead keep sending things as fast as you can.

The default write concern for mongorestore is majority.  This would have naturally throttled the insert rate, allowed the secondary to keep up, prevented the restore from overwhelming the server ... oh and by default it would have used one worker per collection and worked on four collections in parallel at most.   This restore looks like it's of a single collection anyway, but we're talking default one thread, waiting for replication and you ran it with 10 threads waiting for nothing.

So I don't really see evidence that there is any bug in mongorestore - it tried to do exactly what you told it.   I can't tell what exactly was going on with resources on your system since a snapshot in time is hard to interpret, but I would start by never again restoring into a live system with unacknowledged writeConcern.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/fbd02811-c6e3-4db0-b478-0830ccb8d6f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!