MongoDB Changestream

50 views

Skip to first unread message

unread,

Jul 16, 2018, 6:05:24 PM7/16/18

to mongodb-user

Hi all,

I created a microservice which is listening to the mongodb changestream notifications of a collection (i:e - mycollection)

and update the change stream data in another collection (i;e - "yourcollection")

for scalability , I created 10 instance of my application which will read the same changestream notifications.

ex -

Suppose at a time 100 records are updated in "mycollection" collections. so change stream will get notification of the 100 records.

1st instance should read 1 - 20 records from the changestream and update that records in other collection (i:e - yourcollection)

2nd instance should update 21 - 40 records from the changestream and update that records in collection "yourcollection".

3nd instance should update 41 - 60 records from the changestream and update that records in collection "yourcollection".

so .. on...

no two instance will get same duplicte data from the change stream .

is there any way to implement the above requirement.?

Technology used - springboot , mongodb , java

Thannks

Dillip

unread,

Jul 23, 2018, 3:18:13 AM7/23/18

to mongodb-user

is there any way to implement the above requirement.?

Hi Dillip,

In order to process the notifications without any duplication, you would need to design the application/client side to behave this way.

The solution would depend on your environments and requirements. There are a few methods that you could try:

Create a worker pool, where you have only one instance listening to the Change Streams then delegate the task to the worker instances for processing. i.e. batch of 20 etc.
Add an aggregation filter to each of the instances i.e. $match pipeline stage. For example, one instance is only going to handle notification where user has a specific id, etc.

Each of the methods above have their own pros and cons depending on your application use case.