How can I filter data with mongorestore in 3.0x and greater?

[jamieorc]

Now that mongorestore no longer has the --filter option in versions 3 and above, how does one filter data being restored? I'm relying on this feature each night to filter dumped data from one database into other databases. It's critical I find a path forward with this so that I can finish migrating to Mongo 3.x. I've had to revert to 2.6 in the meantime. 

One thought: is it possible to use 2.6 tools with 3.0 dbs? (Though the performance hit would be painful.)

Cheers,

Jamie

[Daniel Doubrovkine]

We've used 2.x tools with 3.x successfully FWIW. The reason were bugs like https://jira.mongodb.org/browse/TOOLS-1039, not --filter though.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/a76f1a2b-35ac-4d5a-9317-379841ad52fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

dB. | Moscow - Geneva - Seattle - New York
code.dblock.org - @dblockdotorg - artsy.net - github/dblock

[Jamie Orchard-Hays]

Thanks, Daniel. The part about possible problems with non-simple Mongo datatypes is worrisome. Did you encounter any issues WRT data?

The way forward for us may be to use mongodump with - -query. The downside is that instead of one dump of everything from which I can import with filters, I’ll have to make numerous—scores of — dumps with --query and then mongorestore from each of those. It’s far less efficient for what we’re doing here. We have nightly process that take the dumped data and import filtered sets from the master set into scores of other databases. 

Jamie

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/uijlrfTRtdk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CALCE7pnLr-yaZ5DK3zkwPxQrfRPSdF7zj_TDMZfyLcO4SUCNaA%40mail.gmail.com.

[Kevin Adistambha]

Hi Jamie,

Now that mongorestore no longer has the —filter option in versions 3 and above, how does one filter data being restored?

The reason --filter is not available in mongorestore 3.0.x and above is because the tools (e.g. mongodump, mongorestore, mongoexport, etc.) were separated from the main server C++ code (e.g. mongod and mongos) and rewritten in Go. Query parsing is part of the server codebase, and is not available as a generic library. For more background on the 3.0 tools rewrite please see Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go.

Pre-3.0.x, the tools share actual mongod server code — that is why options like --filter or --dbpath could exist in mongorestore. With MongoDB 3.0 the tools were rewritten to support the new Storage Engine API and include features like improved concurrency.

The way forward for us may be to use mongodump with —query. The downside is that instead of one dump of everything from which I can import with filters, I’ll have to make numerous—scores of — dumps with —query and then mongorestore from each of those.

For MongoDB version 3.0.x and above, using mongodump --query is the recommended method if you need partial dump of your data.

One thought: is it possible to use 2.6 tools with 3.0 dbs? (Though the performance hit would be painful.)

This is not recommended due to major differences between versions 2.6.x and 3.0.x (e.g. auth schema, WiredTiger, etc). There are also bugs in older versions (e.g. the aforementioned TOOLS-1039) that were fixed in later versions and could compromise your backup.

Best regards,
Kevin

​