Aggregation Pipeline Parallel Execution

[Johan Jaicari]

Hello everyone!

I'm having this problem where I've a sharded environment and an aggregation pipeline to execute.
From the side of the application, I've some microservices which execute this pipeline.

My qyestion is, can I make parallel executions of the pipeline, or should the microservices wait for each others job to finish.

Thank you.

[Stephen Isienyi]

To deduce here, I think you are worried about the I in ACID. Yes, you can make parallel executions of the pipeline in this case.

Each db service request runs in an OS thread dedicated to it by the db server. This will serve as the isolated context in which a particular db request sent by your microservice will be handled in its entirety. So where the db request is an aggregate pipeline, the result of the first stage execution will exist in and remain local only to that particular OS thread. Each remaining stage in the pipeline acts on the results of its predecessor. Keep in mind that all of that is isolated in an OS thread dedicated to handling this request by the db server. 

Subsequent db requests that were sent by other clients of your db server (including this microservice instance and your other microservices) are each assigned any available OS thread by the db server, and the aforementioned process repeats. If no thread is currently available, then incoming db requests are placed in a queue until an OS thread can be assigned the next request in the queue and the aforementioned process starts for that particular request.

So unless the order of completion is important, you can simply invoke all of your microservices. Which ever makes a db request that reaches the db server first and/or gets assigned to the earliest bd server instance in the cluster gets executed first.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/df3c8537-c962-4fbc-a8fb-4cdd66665770%40googlegroups.com.

[Johan Jaicari]

Hey Stephen,

Thank you very much for your quick response. Is what you're saying based on some MongoDB references or documents? If so, please send it to me.

Thanks again!!

To unsubscribe from this group and stop receiving emails from it, send an email to mongod...@googlegroups.com.

[Stephen Isienyi]

The architecture I described Is how databases (or any multithreaded servers) handle requests in parallel. MongoDB is written in C++ which supports multithreading so you can adequately surmise that it utilizes that facility. In fact, it would be alarmingly suboptimal if it didn’t.

It is highly unlikely that any documentation or references exist on something of this nature per se. It is considered internal details of the system and therefore inapposite to the ability to use its API. 

However, you can rest-assure that as long as you define your aggregate pipelines in manner illustrated in mongodb docs for sharded collections and then invoked them through your microservices, your db results per microservice would be the aggregation of the stages in its pipeline and db state at the beginning of the execution of the first pipeline stage.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/fb94db4e-3798-4511-a0e3-d7ceba466039%40googlegroups.com.