TOOLS-23: using tar to stream mongodmp/mongorestore

[Blake Mitchell]

I have a preliminary implementation of TOOLS-23 against current master. The code is here: https://github.com/kalahari/mongo-tools/tree/TOOLS-23 (https://github.com/kalahari/mongo-tools/commit/55d37d789157a6bd96a713e1540866f544355a70, https://github.com/kalahari/mongo-tools/commit/2544816aebeaf72a263c22bee41e983310b7248e)

The general approach was to write to and read from file chunks in a tar archive. Because tar needs file sizes in advance, I used a pair of 4MB buffers to write the dump data in file chunks. The tricky bit was to make sure that mongodump is writing files in the same order that mongorestore needs them. There is slightly more risk, because mongorestore is unable to interrogate the dump ahead of time, and abort if the dump looks malformed, before any restoring starts. On the other hand I have successfully been able to do this:

$ mongodump --db=Test --tar --out=- | bzip2 --fast | ssh blake@restorehost 'bunzip2 | mongorestore --db=Test --drop --tar --dir=-'

And it works!

I would love to get a second pair of eyes on this code before submitting a pull request. There are a couple rough edges: I need to implement users/roles and oplog restore from tar, and add some more comments, but I am very encouraged by what is there now. Any feedback would be much appreciated

Cheers,
Blake


Here is an example of what a tar dump looks like:

$ tar -tvf dump.tar
-rw-r--r-- 0/0            4375 2015-04-24 16:28 Test/system.indexes.bson.000000000000
-rw-r--r-- 0/0             108 2015-04-24 16:28 Test/Jobs.metadata.json.000000000000
-rw-r--r-- 0/0              33 2015-04-24 16:28 Test/Jobs.bson.000000000000
-rw-r--r-- 0/0             108 2015-04-24 16:28 Test/Test.metadata.json.000000000000
-rw-r--r-- 0/0              83 2015-04-24 16:28 Test/Test.bson.000000000000
-rw-r--r-- 0/0             126 2015-04-24 16:28 Test/TestBsonIdCreatorClass.metadata.json.000000000000
-rw-r--r-- 0/0             836 2015-04-24 16:28 Test/TestBsonIdCreatorClass.bson.000000000000
-rw-r--r-- 0/0             114 2015-04-24 16:28 Test/collection.metadata.json.000000000000
-rw-r--r-- 0/0            2963 2015-04-24 16:28 Test/collection.bson.000000000000
-rw-r--r-- 0/0             113 2015-04-24 16:28 Test/importjob.metadata.json.000000000000
-rw-r--r-- 0/0         2457948 2015-04-24 16:28 Test/importjob.bson.000000000000
-rw-r--r-- 0/0             118 2015-04-24 16:28 Test/importproperty.metadata.json.000000000000
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/importproperty.bson.000000000000
-rw-r--r-- 0/0         4198164 2015-04-24 16:28 Test/importproperty.bson.000000000001
-rw-r--r-- 0/0         2891511 2015-04-24 16:28 Test/importproperty.bson.000000000002
-rw-r--r-- 0/0             378 2015-04-24 16:28 Test/orgdata.metadata.json.000000000000
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000000
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000001
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000002
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000003
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000004
-rw-r--r-- 0/0         4227072 2015-04-24 16:28 Test/orgdata.bson.000000000005
-rw-r--r-- 0/0         2820859 2015-04-24 16:28 Test/orgdata.bson.000000000006
-rw-r--r-- 0/0             113 2015-04-24 16:28 Test/system.js.metadata.json.000000000000
-rw-r--r-- 0/0              66 2015-04-24 16:28 Test/system.profile.metadata.json.000000000000
-rw-r--r-- 0/0          167948 2015-04-24 16:28 Test/system.profile.bson.000000000000