$lookup aggregation on a dynamic number of collections

[jan...@princeton.edu]

Hello! I am working on a project involving multiple collections in MongoDB, and I want to aggregate the data into one massive collection. Every collection has a field "barcode" with the barcode of relevant products, so I have been trying to use $lookup and aggregate to bring the collections together. The problem is that the number of collections is dynamic, so I cannot simply call $lookup multiple times (unless there is a way to iterate through a loop during a call to aggregate). I initially aggregate the first two collections to a new collection, then call db.collection.aggregate on the newly created collection, so the data from the first two collections is stored.  When I try to map through each remaining collection (their names are stored in an array), I call db.collection.aggregate on the newly created collections and set $out to the new collection itself so that the data compounds and theoretically will continue to grow. The lookup works, but it overrides the previous output, so the only data that is stored is from the last queried collection. Let me know if there is a way around this problem, or if you need more clarification. Thanks!

[Kevin Adistambha]

Hi,

The problem is that the number of collections is dynamic

Could you provide a more concrete example of what you’re trying to do? Some document examples, workflow description, example results, and your MongoDB version would be useful.

When I try to map through each remaining collection (their names are stored in an array)

What is the significance of this array and where is it stored? Is it in a document somewhere, or is this internal to your application?

I initially aggregate the first two collections to a new collection, then call db.collection.aggregate on the newly created collection, so the data from the first two collections is stored.

If I understand correctly, you’re using $out multiple times to “collect” the $lookup results into one output collection. Is this correct? If yes, this will not work as expected since $out will overwrite an existing collection with the same name.

Best regards
Kevin

​