How MongoDB can be used as a message broker ?

[aliane abdelouahab]

Hi,

I was searching on internet trying to understand the concept of RabbitMQ, and then found people replacing it with Redis, but other results were using MongoDB,

can you make it  clear for me please, I dont get it (maybe i dont understand the concept of message broker).

[s.molinari]

A message broker, is basically a queue of activities or communication in a networked system, in order for that activity or communication to be "controlled" or stored for a better time to execute them in other parts of the system. I can't imagine Mongo or Redis being better at it than RabbitMQ, because it is specifically built for messaging. But then again, my knowledge in this area is limited. I can imagine if you need a very simple queue, Mongo could actually be quite good at that. 

Scott  

[aliane abdelouahab]

Thank you,

and then what i would save? i will use a push and pull all the time?

[s.molinari]

If you can do your messaging queue over Mongo, you'd just be saving yourself having to learn and use a second technology. 

Scott

[aliane abdelouahab]

With MongoDB and Tornado, there is an asynchronous driver which is Motor, so it will be benefic pour that, but what i will save as data?

[Rob Moore]

Aliane -

First - Yes, you can have MongoDB act like a message broker. I even did a presentation on what is needed (and a little on what a broker does) at the MongoDB DC 2013. You can find the slides here and the example code here. 

The idea of using messaging within a system is to allow the components/applications/processes within the system be decoupled and scaled independently. The broker acts as the central point that all of the components connect and communicate through. 

If you use a queue then one part of the process slows down or has a spike in requests it is the job of the message broker to absorb (accept) and hold the requests until the component processing the requests can catch up. Using publish-subscribe (aka: topic, fanout exchange) then the sending process can notify multiple other processes without having to know about all of the components that want to know about the message. There are other, more advanced, patterns such as dynamic scaling[1], but I would avoid doing those until you are comfortable with the basic topic and queue semantics. Note that while the message broker is the "central" point that does not mean it is a single point. Brokers like RabbitMQ have very good clustering capabilities.

Why would you use MongoDB (or Redis)  as a message broker? I think the only reasonable answer to that is when the use of a messaging pattern is very limited within the application. If that is the case the cost of having another piece of infrastructure that you have to install, understand, monitor, repair, and upgrade is higher that the cost of writing and maintaining code to force your datastore into a role it was not really designed to support. If messaging is going to be a central part of your application then the scale tips (quickly) in the other direction and I would use a purpose built application like RabbitMQ (which I have used and like).

HTH -

Rob.

1: A process watches the pending messages on queues and starts/stops processes as demand increases/decreases.

[aliane abdelouahab]

Thank you :)

I just checked the codes, and it seems that the content of the message are kind of low level OS ? i thought it was up to the dev to send the content the message, but it seems that it is not? (something like: take the function lambda and save its contents and retrieve it later...)