Move document from one collection to another on the same database

[Senor Cardgage]

I need to move select documents from one collection to another collection on the same database. I do not want to pull the documents from the database to my application and then push them straight back to the same database. It seems there must be a more efficient way to handle this scenario without the documents needing to come across the wire.

I thought about using a stored javascript function on the server that would simply move the records across and then call it using eval(), but all advice seem to indicate using stored javascript functions is a "Bad Idea"
(like the note at the top of this page: http://docs.mongodb.org/manual/tutorial/store-javascript-function-on-server/

)

What other approaches would you recommend for handling this?

[s.molinari]

I'd also like to know the answer to this, just out of curiosity.

I'd also think, how often this process has to happen and the approximate size of the document or documents being copied, might be good information to know to answer the question?

Scott

[Mirko Bonadei]

Why not doing this from the shell of your mongod?

You don't need a stored procedure for this. For example:

var documentsToMove = db.collectionA.find({<YOUR_FILTER_HERE>});

documentsToMove.forEach(function(doc) {

    db.collectionB.insert(doc);

    db.collectionA.remove(doc);

}

I hope this helps. 

Cheers,

Mirko

[Senor Cardgage]

This needs to occur at any time as triggered by the user of my application.  So, I need the ability to trigger this behavior at any time from within my application. I could use eval() from the application, but that doesn't seem any better than using a stored javascript function.

[Mirko Bonadei]

Oh OK. But If I were in you I will do it from my application. Find -> Insert -> Remove.

It is the best way. Running evals is not a good deal for me. "By default, eval takes a global write lock before evaluating the JavaScript function. As a result, eval blocks all other read and write operations to the database while the evaloperation runs. Set nolock to true on the eval command to prevent the evalcommand from taking the global write lock before evaluating the JavaScript. nolock does not impact whether operations within the JavaScript code itself takes a write lock.". Some drivers doen't have nolock and by the way it is not a good way of interacting with mongo for the other warnings showed up on the manual page.

Why you don't want to do it from your application? Which is the constraint?

Mirko

[Asya Kamsky]

While this works, as does running the equivalent code in the
application, keep in mind that the "insert" and the "remove" are *not*
atomic. This means that if your application/code crashes or for some
other reason terminates between the insert and the remove, there will
be a copy of this document both in the from and the to collections.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/cc7a1447-47ac-4ac7-bab4-35ebd42490df%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Mirko Bonadei]

Sure, maybe it is a better idea to do it from your application with a Job in queue which is considered completed only if all the things are done.

I don't think that MongoDB provides an atomic way of doing it.

Mirko

[Senor Cardgage]

What I'd really like to know is whether there is a recommended way of moving the records *at the server* without pulling them across the wire and then pushing them back. A user of my application may approve up to 1000 records at a time which is the stimulus for moving them to another collection. It would be so much more efficient if I didn't have to transfer the records in two directions across the wire.

[Mirko Bonadei]

I have not a direct experience with this kind of situation.

If I were in you I would try with eval but with nolock set to true. Let us know.

Mirko

[s.molinari]

Could it be you might be looking for a solution that is completely wrong for the problem? Could you explain in better detail what happens, when "approvals" happen? Maybe a proper/ better pattern/ method could be found for you.

Scott

[Joel Harris]

Records are created in the "Pending" collection. These are created by automated processes. My (web) application shows these pending records to users who approve them or decline them. Approved records are moved to the "Approved" collection. Declined records are moved to the "Declined" collection.

When displaying pending records to the user, the records are not read in their entirety into the application. Only those fields required for the approval process are read.

On Mon, Mar 31, 2014 at 9:40 AM, s.molinari <scottam...@googlemail.com> wrote:

Could it be you might be looking for a solution that is completely wrong for the problem? Could you explain in better detail what happens, when "approvals" happen? Maybe a proper/ better pattern/ method could be found for you.

Scott

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/qB4jsb8UxQ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e279fe12-0f5e-4ebf-8706-aa975bf54cb8%40googlegroups.com.

[Mirko Bonadei]

Why don't use a status field in each document? With a proper index obviously.

status: Pending|Approved|Declined.

If you want you can remove declined and use powerOf2Sizes to help MongoDb in space optimization.

Mirko

[William Zola]

Hi Senor!

One of the things that is sometimes hard to understand about MongoDB is that there IS NO "at the server".  :-)

As a general design principle, MongoDB does not support any operation which cannot be efficiently performed on a sharded cluster.  If you think about it, once you're running in a sharded cluster, the very concept of *THE* server goes away -- you have multiple servers.  Dozens of them, in the most general case, even if you only consider Primary nodes.

As a result, many operations which would be simple or trivial to perform "at the server" become intractable, since distributed operations are hard to perform efficiently with full generality.  

A rhetorical question for you -- how do you know that pulling things across the wire and pushing them back will be a bottleneck?  Have you measured how long it takes?  Or are you just relying on your intuition?  Maybe -- just maybe -- it isn't as big of a performance penalty as you think it is.  

To answer your direct question: the only way in MongoDB to move a document from one collection to another at the "server-side" is via stored Javascript.  This is not recommended -- it will take the global lock and block all other operations, and it Simply Will Not Work if you have to shard either one of the collections.  

As others have said, it's probably most efficient to change your schema so that all of these documents are in one collection.  

If you choose not to change your schema, and If your measure network latency turns out to be significant (50 ms or more) then you can optimize this process by having a dedicated client program running on the same machine as your 'mongod' (or one of your mongos) which performs the find()/save()/remove() operation.  

Let me know if you have further questions.

 -William 

[Joel Harris]

Thanks William. Your response was very useful to me in deciding which avenue to persue.

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/qB4jsb8UxQ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/bbefcf76-62cd-4ef2-bc29-be63c826df12%40googlegroups.com.