RabbitMQ as communication server for mobile devices

[Michal Levý]

Hi everyone

I have following scenario:

1) Many Android clients (from 1000 to 10 000 clients) - not connected all the time (messaging seems like good fit for this)

2) Each sending max 100 messages\day

3) Also needs to send some messages back from server but not very frequently (10\day)

I was looking on rabbit tutorials, AMQP spec etc.

It looks like collecting data from androids is easy task. All will send data into single exchange\queue.

But what about sending messages to Androids from server?

1st i was thinking about having single queue and to subcribe from each device with some kind of "filter" (based on some header field). I know there are message brokers which support this scenario.

But it seems Rabbit doesn't support this. Do i really have to create one "sending" queue for each client (using direct exchange with proper "per device" routing key) ?

How much overhead queues in rabbit have ?

It can be also management issue and really feels like overkill when number of messages send from server to device is really low....

Any advice will be appriciated.

Thanks

Michal

[Michal Levy]

Hi Tim

On Thu, Nov 15, 2012 at 9:58 AM, Tim Watson <t...@rabbitmq.com> wrote:

Hi Michal,

On 11/14/2012 04:49 PM, Michal Levı wrote:

Hi everyone

I have following scenario:
1) Many Android clients (from 1000 to 10 000 clients) - not connected all the time (messaging seems like good fit for this)
2) Each sending max 100 messages\day
3) Also needs to send some messages back from server but not very frequently (10\day)

I was looking on rabbit tutorials, AMQP spec etc.
It looks like collecting data from androids is easy task. All will send data into single exchange\queue.
But what about sending messages to Androids from server?
1st i was thinking about having single queue and to subcribe from each device with some kind of "filter" (based on some header field). I know there are message brokers which support this scenario.
But it seems Rabbit doesn't support this.

It isn't hard to set this up in Rabbit! You will need to think about solving the problem in AMQP terms though, which is admittedly a bit different from what you'd do with say JMS or whatever.

Im trying ;) ...but its still new concept for me...

 

 

Do i really have to create one "sending" queue for each client (using direct exchange with proper "per device" routing key) ?

As a matter of fact, clients never send messages directly to queues, but rather send them to exchanges (as you've already noted), after which they're routed to zero or more queues based on the bindings you've set up (between the exchanges and queues). So for example, you can send a message to an exchange from one client and receive it on another a la pub/sub by using a topic exchange. You can apply content based routing using the headers exchange. If you've got data that needs to go out to the devices then you first need to figure out what your filters look like. Once these are in place, you can use one of the built in exchange types to place the messages into the broker's care. Each device that wishes to consume data can either create a temporary queue and bind it to the exchange(s) in the right way, or connect to an existing queue which is bound to a topic exchange for some set of keys that interest you.

Yep, i know you always send message to exchange and not to queue directly (and i think its very nice concept).

My question was more about "consuming part" - when i have messages for one and only one device\client, only way to get this message into device is to create queue for each of them (and using proper exchange types\binding) - because there is not something like "subscribe to messages from queue xxxx with header property DeviceId = 564165434" in Rabbit...

And you answered my question ;) ...i have to have one "incoming" queue for each device.

Temporal queues created by devices are not option for us. We work with same device for a long time (say few months) and we need to send messages to devices even when they are not connected atm..

As i understand how exchanges works, if there is no queue bound to exchange with correct routing key, messages are just dropped (or can be returned back to publisher)...

 

How much overhead queues in rabbit have ?

You can have lots (i.e., thousands) of queues without suffering too much, though there is obviously *some* overhead (memory, etc) and its worth planning your topology carefully so as to get the best out of Rabbit. We can help with that (here) by pointing you to the right resources and documentation, or if you want elaborate on some of the finer details of the design we often have lively and interesting discussions about how to solve various problems on this mailing list!

I have made some tests and 1 durable queue created on rabbit running on 64bit Debian takes cca 10kb of memory. Thats pretty acceptable imho

"Design" is probably too strong word for what i have now :) ....just thinking...

Going back to something you mentioned at the beginning of your post:

It looks like collecting data from androids is easy task. All will send data into single exchange\queue.

This really depends on what the whole messaging topology needs to look like. Sometimes you need to propagate messages into various places, and building blocks like fan-out and exchange-2-exchange bindings can help with this. Other times, it actually makes more sense to send some messages to a specific exchange and others to, well, another. As you're making design decisions such as these, you'll want to consider various factors which include performance and resource consumption on the broker, scaling up/out (as you add more clients, potentially cluster the broker and so on) and of course the management of all this infrastructure, from the perspective of both the client application(s) and the administration of the broker itself.

Yep, i'v oversiplified here.

Im planning to have different exchanges\queues for data messages send from devices independently, separate for responses for server requests and different for logging etc...

Not set in stone yet but im starting to see the light ;)

 

It can be also management issue and really feels like overkill when number of messages send from server to device is really low....

It's actually quite easy to set up a temporary, auto-delete (once the client disconnects) queue that is used simply to bind to an exchange that you're interested in. The semantics probably need some thought however: how are you going to consume messages (e.g., round robin, fan-out, etc), how will you know to delete them from the broker (using acks, etc) and so on. This is where you need to think carefully about your design. Either way, there shouldn't be much management to do if you're using temporary queues to make consumption and guarantee that the queues are deleted once you're finished with them (i.e., the client goes away). If you're using long lived (shared) queues then the design complexity pushes back into the exchanges and bindings you configure, but that's done just once and then clients need to know where/how to connect.

Any advice will be appriciated.

If you've been through the tutorials, then hopefully it's clear that consuming from a queue is relatively simple and that you have a number of choices when it comes to 'filtering' what you receive. If you want to go into a bit more detail about these 'filters' then we can look at which topologies might best support your use cases.

When i was talking about 'filters' it was really meant as "how to subscribe to sigle queue and receive only some messages based on some condition", not how to distribute messages into different queues based on exchange type\routing key really :)

Thanks very much for your time Tim!

Cheers

Michal

 

Cheers,

Tim