Combining documents from two separate collections. Duplicate ids issue.

[kumar]

Hi All,

I am going to combine documents from two separate collections.
I will do using mongodump and mongorestore. commands.

There may be a chance of having same auto generated Id in both
collections.
Will combine operation (using mongo restore) happen properly due to
same auto generated Id?

Thanks.
Kumar

[Raxit Sheth]

Best is to handle via application side logic instead of below commands.

Kumar

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Raxit Sheth]

http://wrtsprt.wordpress.com/2011/08/17/merging-collections-in-mongodb/

you may want to chk comment also on this blog

[Adam C]

If you are planning on having the field that may have duplicates be unique (have a unique index), then you need to decide how you want to handle that case.  The most obvious way is not to have a unique index on that field, reconcile the differences etc. but it is really up to you.  See this for example:

http://www.mongodb.org/display/DOCS/Indexes#Indexes-dropDups%3Atrue

Alternatively you may want to have more control over what happens, in which case you might want to explore writing something yourself and using application code to reconcile your data before insertion (if you do look at the suggested merging collections link, be sure to take a look at Kristina's comment about batch insert).

Adam

On Saturday, May 12, 2012 10:14:43 PM UTC+1, Raxit Sheth <Mobile 4 Mumbai> wrote:

http://wrtsprt.wordpress.com/2011/08/17/merging-collections-in-mongodb/

you may want to chk comment also on this blog

On Sun, May 13, 2012 at 2:39 AM, Raxit Sheth <raxitsh...@gmail.com> wrote:

Best is to handle via application side logic instead of below commands.

On Sun, May 13, 2012 at 2:31 AM, kumar <kumar...@gmail.com> wrote:

Hi All,

I am going to combine documents from two separate collections.
I will do using mongodump and mongorestore. commands.

There may be a chance of having same auto generated Id in both
collections.
Will combine operation (using mongo restore) happen properly due to
same auto generated Id?

Thanks.

Kumar

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.

[jeannot]

Hi,
I have the same problem. In my case, i'd if a document has an _id
already present in the collection, i'd like it to be regenerated to
make sure all documents are inserted. Is there an option for that? if
not, how can I remove the _id fields of dumped documents?
thanks
jean

On 14 mai, 14:07, Adam C <ad...@10gen.com> wrote:
> If you are planning on having the field that may have duplicates be unique
> (have a unique index), then you need to decide how you want to handle that
> case.  The most obvious way is not to have a unique index on that field,
> reconcile the differences etc. but it is really up to you.  See this for
> example:
>
> http://www.mongodb.org/display/DOCS/Indexes#Indexes-dropDups%3Atrue
>
> Alternatively you may want to have more control over what happens, in which
> case you might want to explore writing something yourself and using
> application code to reconcile your data before insertion (if you do look at
> the suggested merging collections link, be sure to take a look at
> Kristina's comment about batch insert).
>
> Adam
>
> On Saturday, May 12, 2012 10:14:43 PM UTC+1, Raxit Sheth <Mobile 4 Mumbai>
> wrote:
>
>
>
>
>
>
>
>
>
> >http://wrtsprt.wordpress.com/2011/08/17/merging-collections-in-mongodb/
>
> > you may want to chk comment also on this blog
>

> > On Sun, May 13, 2012 at 2:39 AM, Raxit Sheth <raxitsheth2...@gmail.com>wrote:
>
> >> Best is to handle via application side logic instead of below commands.
>

> >> On Sun, May 13, 2012 at 2:31 AM, kumar <kumar.a...@gmail.com> wrote:
>
> >>> Hi All,
>
> >>> I am going to combine documents from two separate collections.
> >>> I will do using mongodump and mongorestore. commands.
>
> >>> There may be a chance of having same auto generated Id in both
> >>> collections.
> >>> Will combine operation (using mongo restore) happen properly due to
> >>> same auto generated Id?
>
> >>> Thanks.
> >>> Kumar
>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "mongodb-user" group.
> >>> To post to this group, send email to mongod...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> mongodb-user...@googlegroups.com.

[Adam C]

There is no automatic way to do this.  There are several of approaches you could take though, for example:

You could restore the data to a new collection, identify the dupes, and insert them without the _id field specified, that would mean a new _id would be generated automatically.  Then you can simply drop the dupes when you do your normal restore as I mentioned.

Or, you could write a script to iterate through the records check for duplicates against your target collection, if found, generate a new one, insert, otherwise insert as-is

Or, you could remove the unique index on the collection, restore all the data, update the dupes in place and re-create the index

It's all about finding the right approach to fit your needs.

Adam

> >>> mongodb-user+unsubscribe@googlegroups.com.

[jeannot]

Hi Adam,
Thanks for your answer, the first solution is ok for my problem
jeannot

> > > >>> mongodb-user...@googlegroups.com.

[Scott Hernandez]

If you simply insert and there is existing document then you will get
an error and you can re-insert with a new _id value, or none, for that
document. To do this, and get back the error, you must do safe inserts
and handle the error case.

> See also the IRC channel -- freenode.net#mongodb