cloneCollection with query and authentication

[James Crosswell]

I need to copy some of the documents from a database/collection on a remote server down to a database/collection with the same name on my local host.

At first glance the db.cloneCollection(from, collection, query) command looks perfect for this as it allows you to specify a query parameter (which would give me the documents that I want). However I see that it doesn't yet support authentication (and my remote server requires authentication)...

On the other hand, none of the alternatives that I can see such as db.copyDatabase(fromdb, todb, fromhost, username, password)or db.collection.copyTo(newCollection) let you specify anything analogous to a query parameter so there's no way to limit the data that would be pulled down in those cases (and the collections on the remote server are too big to practically pull down all of the data). 

Is there a solution to this that I'm not seeing?

Cheers,

James

[Will Berkeley]

Hi James. I think a good option here is to use mongodump from your target machine to get a BSON file dump of the documents that you want and then use mongorestore to put the documents into the target MongoDB. mongodump does accept a json file with a query in it to limit what documents are pulled down.

-Will

[James Crosswell]

Thanks Will... I guess that could do what I want (albeit not very elegantly). I actually want to merge the documents from the remote database into collections with the same name on a local database. I see mongorestore has a -drop parameter but I don't want to drop the local collections entirely... I'm more wanting to do an upsert. So I guess I'd have to:

1. Run mongodump against the remote database with the query parameter (so that I only get the documents I'm interested in)
2. Run mongorestore to restore the data to a temporary database on the local server
3. Maybe run a javascript script file to copy across each of the collections from the temporary database (upserting the documents into the target database)
4. Drop the temporary database

Not ideal, but workable. It's a pity cloneCollection can't accept any authentication parameters but I guess the mongodb crew have bigger fish to fry!

Cheers,

James

[Will Berkeley]

You can feed the cloneCollection authentication fish by watching/upvoting SERVER-2931 :)

-Will

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9ad764af-aba5-41af-8b90-37026f0e63b2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Asya Kamsky]

What exactly does "merge" mean?  If the document doesn't exist, you want to insert it, but what if it does exist?

The option --drop is only if you want to drop the target collection before restoring.  By default the collection will be left alone and new documents will be added to it.  Now, if you have documents with already existing _ids, then the original document stays and the new one is not restored.

If that's the behavior you want then that's exactly what mongorestore already does.

Asya

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

[James Crosswell]

Hi Asya,

Thanks, but not quite what I need... I need an upsert.

Cheers,

James

[Asya Kamsky]

An upsert is insert if it's not there and update if it is.

How would it update the document if it's already there?  Would it update it to be equivalent to what's being restored?

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ade1815b-12e4-46de-bc72-254e23cdb4b4%40googlegroups.com.

[James Crosswell]

Overwrite.

[Asya Kamsky]

Sounds like this ticket https://jira.mongodb.org/browse/SERVER-2913 would provide the functionality you want

Meanwhile, the only workaround I can think of (if the database this collection is in is not too big, and the collection itself is not too big) you can use copydb command which does take auth for remote DB and then do a cloneCollection with query locally.

Of course, that's assuming you want to use some database built-in command.  You could also write a small script which authenticates, queries for desired documents and upserts them into the target collection.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/49c066b0-dd22-4e28-ada2-9c70cf70352e%40googlegroups.com.

[James Crosswell]

Hi Asya,

Thanks. I ended up going the second route (writing a small script), which works OK. 

Cheers,

James