Using mongorestore for oplogreplay with --numInsertionWorkersPerCollection

[chetan sharma]

If I use mongorestore with numInsertionWorkersPerCollection > 1, it doesn't bring any performance improvement. I have tried with 5, 10, 16 as a value and I couldn't see any significant improvement. I have a 8 core machine with 64 GB RAM and my complete oplog size is around 1 GB (around 1 million request on same collection).  So I don't think hardware is the limitation here. Kindly let me know what could be the reason behind it.

Basically, I was just comparing the mongorestore with  sync (which is used to update oplogs on secondary). In case of sync, we have default 16 workers which can apply the oplogs concurrently and I was hoping I can do same with mongorestore too.

Thanks in advance :)

[Kevin Adistambha]

Hi Chetan

If I use mongorestore with numInsertionWorkersPerCollection > 1, it doesn’t bring any performance improvement.

From my understanding, you are trying to do mongorestore --oplogReplay --numInsertionWorkersPerCollection x. Is this correct?

The setting --numInsertionWorkersPerCollection will affect how many goroutines are used in restoring a regular collection (https://github.com/mongodb/mongo-tools/blob/v3.4/mongorestore/restore.go#L271-L274)

In contrast, that setting doesn’t have an effect on restoring the oplog when you specify --oplogReplay. This is because mongorestore restores the oplog in a serial manner, since the oplog is not a regular collection (https://github.com/mongodb/mongo-tools/blob/v3.4/mongorestore/mongorestore.go#L476-L482).

Note that I am assuming you are using mongorestore version 3.4.

Best regards,
Kevin

​