Maximum number of binding keys for a binding

[Elias Mårtenson]

In my application, I have messages sent an an exchange with a routing key that contains very detailed information about the actual message being sent. The subscribers can then use a very specific filter to retrieve only the exact messages it wants. If it needs another type, it just adds another subscription between the exchange and its queue.

However, in some cases (and especially for a long-running consumer) the list of subscriptions can be very large. It could possibly reach tens of thousands.

My question is: Is there a practical limit to the number of binding keys I can add to a subscription? Is 10000 binding keys reasonable?

Regards,

Elias

[Michael Klishin]

On 2 January 2015 at 07:54:28, Elias Mårtenson (lok...@gmail.com) wrote:
> My question is: Is there a practical limit to the number of binding
> keys I can add to a subscription? Is 10000 binding keys reasonable?

You certainly can have 10K bindings. But the actual performance of such system depends on the exchange type used.

E.g. fanout exchange routing is pretty obviously O(n) while topic
exchange is trie-based [1][2] and scales significantly better.

Note that with 10s of thousands of consumers having a queue each you'll need to keep an eye
on how much RAM a node uses. But you can always use more than 1 node and spread connections between
them. 

1. http://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/
2. http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Elias Mårtenson]

On 2 January 2015 at 18:02, Michael Klishin <mkli...@pivotal.io> wrote:

On 2 January 2015 at 07:54:28, Elias Mårtenson (lok...@gmail.com) wrote:
> My question is: Is there a practical limit to the number of binding
> keys I can add to a subscription? Is 10000 binding keys reasonable?

You certainly can have 10K bindings. But the actual performance of such system depends on the exchange type used.

Thanks a lot for your reply.

I'm using a topic exchange with a small number of queues but many bindings. See my clarification below.

 

E.g. fanout exchange routing is pretty obviously O(n) while topic
exchange is trie-based [1][2] and scales significantly better.

I believe this should alleviate my fears. I assumed that it walked through all the topic bindings in sequence to determine whether the message should be delivered to the queue. It seems as though this is in fact using something much more clever?

 

Note that with 10s of thousands of consumers having a queue each you'll need to keep an eye
on how much RAM a node uses. But you can always use more than 1 node and spread connections between
them.

In this case, I only have a handful of consumers (perhaps 10 or so). Each consumer has its own queue, but each one of these queues can have tens of thousands of bindings to the exchange.

In other words, with bad ascii graphics:

    Input -> Exchange |--> 10k bindings -> Queue_1

                      |--> 10k bindings -> Queue_2

                      |--> 10k bindings -> Queue_3

                        ......

                      |--> 10k bindings -> Queue_10

Regards,

Elias 

[Michael Klishin]

On 2 January 2015 at 13:12:52, Elias Mårtenson (lok...@gmail.com) wrote:
> In this case, I only have a handful of consumers (perhaps 10 or
> so). Each consumer has its own queue, but each one of these queues
> can have tens of thousands of bindings to the exchange.
>
> In other words, with bad ascii graphics:
>
> Input -> Exchange |--> 10k bindings -> Queue_1
> |--> 10k bindings -> Queue_2
> |--> 10k bindings -> Queue_3
> ......
> |--> 10k bindings -> Queue_10

It should work fine. Run some basic benchmarks and see for yourself ;)