Adding data from temporary collection into existing collection

[Soorya Prakash]

Hi, 

  I am having the usecase where I need to do multiple write operation in the collection and collection will increase to millions of record.Since I was doing all the write operation in a single large collection, the write performance was slow and also using index on that.Instead of writing in the single larger collection, I f I write in the temporary collection and then I move it larger collection using the below query,  

Example :  Collection A with 100k record

                 Collection B with 1 million records

Collection A -> Collection B(with indexes).

Will it be quick to copy into the larger collection?

There will be any performance degradation or like blocking in the Larger Collection?

$out is replacing completely the collection.Is there way to add it without replacing it?.

If its need to inserted by looping then I can use directly in the existing collection right?

Thanks

Soorya

[Lungang Fang]

hi Soorya,

Since I was doing all the write operation in a single large collection, the write performance was slow and also using index on that.

May I know what led you to believe of the slow performance? I.e. metrics and tools you used. Collection size itself usually does not incur performance degradation.

There are a number of factors that may impact performance, such as inappropriate indexing strategies, hardware availability, or the number of open database connections.
For more performance related topics, see Analysing MongoDB Performance. You can also utilise mtools to help with slow queries analysis.

Instead of writing in the single larger collection, If I write in the temporary collection and then I move it larger collection using the below query, … Will it be quick to copy into the larger collection?

Depending on your use case, as you still need to consolidate the temporary collection into the large collection eventually, this may not be a good solution. You need to figure out the root cause of the performance issue.

$out is replacing completely the collection.Is there way to add it without replacing it?.

$out always replace the whole collection.

Regards,
Lungang

​