RabbitMQ topic performance with many binding keys

[Rod Hughes]

I posted this at the pivotal.io support but haven't received a response yet, so hoping someone here can provide some input.

We are considering using RabbitMQ for our product but need to know if it should perform well for our use case.

In our use case, a user is associated with a group.  A group’s users are typically served by a single server, but can be served by multiple servers at times due to rebalancing, temporary network outages, etc.

The servers retrieve the user and group data from an external database.  When the data for a user or group is changed, the servers must be notified of the change so they can update themselves (e.g. retrieve the new/updated data from the database or delete the data as applicable).  The servers use pub/sub for this. When the database is updated for a change, a message is published to a topic. The servers subscribe to the topic and then receive these messages.

If there is only a single topic however, this means a large percentage of the messages a server receives are discarded as the user/group being changed is not currently served by the server.  This wastes both network and processing resources.

For example, say we have the following:

1,000,000 users

10,000 groups (average of 100 users per group)

20 servers (average of 50,000 users/500 groups per server)

1000+ messages/second on average

A particular server may only serve groups 1-500 for example and hence only need messages for those groups.  When the database is updated for a change, a message could be published to a topic such as update.group1 or update.group2.

With RabbitMQ and a topic exchange, the server could use binding keys such as *.group1 - *.group500 or #.group1 - #.group500 to receive only messages for the groups it is currently hosting.

Would this perform well given there would be 10,000 binding keys with each of the 20 servers having 500 binding keys?

I assume the servers can dynamically add and remove binding keys as necessary (e.g. in our use case as groups are loaded to/unloaded from a server)?

Thanks!

[Rod Hughes]

BTW...per the blog posts I found at https://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/ and http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/ it seems like RabbitMQ should be able to perform well when there a many binding keys, but it would be nice to have some confirmation of this.

[Luke Bakken]

Hi Rod,

I posted this at the pivotal.io support but haven't received a response yet, so hoping someone here can provide some input.

Sorry you haven't gotten a response. I just subscribed to the RabbitMQ topic on that site. For what it's worth, RabbitMQ core engineering most closely monitors this mailing list, then other places like Stack Overflow and the pivotal.io community site. 

For questions like yours, the standard response from the RabbitMQ engineering team is "run benchmarks using your expected workload". You don't mention your expected message rate, nor expected latency, nor how often bindings will be added and removed - all of which could have an effect on performance. Will you be using a cluster and mirroring queues? Again, that affects performance. Finally, on what hardware or VM will you be running your system?

At this point it would be a complete guess to say "sure, RabbitMQ would be A-OK in your environment".

PerfTest is a great tool to use for benchmarks - https://www.rabbitmq.com/java-tools.html

Thanks,

Luke

[Rod Hughes]

Hi Luke,

Thank you for the response and information.

Yes, there is definitely some vagueness in the details.  I will expand on them below, but the root question is more whether a large number of binding keys (10,000 in the example) but with a common pattern such as *.group<X> or group<X>.* would by itself be a general issue for RabbitMQ.  Reading those blog posts it seems RabbitMQ should be able to handle this well, but if there a known issue it would of course be very useful to know in advance.

Additional details:

Message rate - Stated 1000+/s on average in the original post, but lets say 3000/s for a little buffer.  Message size will be small as well (typically under 1k).

Expected latency - Latency is not extremely important (don't need sub-second times).  Within 15s should be fine.

Bindings added/removed - On a stable system will be infrequent.  When a server starts, bindings will be added as groups are onboarded but rate should still be low.  A quick blast of say 100/s after startup could be possible (generally would be much lower) but would quickly be gone as the server becomes stable again.

Cluster and mirroring queues - Yes...HA is required.

Hardware/VM - TBD...performance testing will help determine what is required to meet the expected load.

Regards,

Rod

[Luke Bakken]

Hi Rod,

We don't know of issues that would affect your scenario. Again, benchmarks are highly recommended.

Thanks,

Luke

[Rod Hughes]

Thanks again Luke!