mongoexport performance

[Mohsin Patel]

hi,

Please can someone help...

we have a 21GB database that is taking around 3 hours to export - does anyone have any suggestions to speed this up?

Thanks

[Eoin Brazil]

Hi Mohsin,

mongodump is a better tool to export production databases in terms of speed as well as creating higher fidelity data (it uses BSON rather than approximating to JSON).

Is this is part of a backup strategy and speed is a concern for you, then using file system snapshots might be an option to speed up the process for you (see http://docs.mongodb.org/master/tutorial/backup-with-filesystem-snapshots/).

Is this part of a backup / regular procedure or is it a one off export ?

Thanks!

Eoin

[Mohsin Patel]

Hi Eoin,

Its not part of backup strategy (we use snapshots for this) - its an extract that we take for import into a data warehouse - unfortunately we are having to take full collection exports. 

Its a nightly process but obviously as data grows 3 hours will soon become 6 hours so i am trying to nip this in the bud.

In terms of using mongodump - does the import method change greatly if going to another RDBMS - namely oracle.

Thanks

[s.molinari]

Do you need every bit of the database data for the data warehousing? I would imagine it is for some sort of BI analysis? 

If not, you could be a bit more selective with the data you export to shorten the export time.

Scott

[Eoin Brazil]

Hi Mohsin,

Scott is correct if you can be selective with your data export, you could use the Aggregation framework to quite a bit of report generation (and summarising) internally within MongoDB and there are a number of BI tools that integrate directly with MongoDB so using one of these might be a better approach for your workflow if it is possible. If you use a Hadoop based analysis you could consider the MongoDB Hadoop connector (http://docs.mongodb.org/ecosystem/tools/hadoop/).

Hope this helps.

Eoin