RabbitMq exchange performance

[Michael G]

Hi,

is it better regarding performance to create multiple exchanges? For instance, if I have multiple file types like zip, json, xml,…

1. Is it better to create one file exchange (Topic exchange), which accepts all files and map them to the right queues. Here, my concern is that if I get many zip files and the messages accumulate at the file exchange less frequently messages are processed late.
2. Or to create one exchange per file type (Zip exchange, Json exchange, Xml exchange).

How, does an exchange processes messages? Is there only one process, which handles the mapping between the exchange and the queues or are the multiple processes, which do the mapping in parallel?

[Karl Nilsson]

The exchange logic is (typically) executed in the channel process (one process per channel create by the client library) and is just a means of calculating which queues should receive a particular message (in short it is done in parallel for every publishing channel in your system). There is no performance benefit in creating multiple exchanges but you may want to do so for domain separation concerns and being able to inspect metrics based on different parts of your system. In your case I'd use a topic exchange with an routing key based on your file type. This allows you to bind more queues for individual file types if they have different processing times.

Adding more queues and consumers is typically the way forward for increasing system throughput.

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Pivotal/RabbitMQ

[Michael Klishin]

Topic exchange routing specifically is unlikely to present a problem unless you have *a lot* of bindings because

it is based on a trie [1][2].

Individual queue throughput [3] and data locality [4] are much more likely to matter most.

1. https://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/

2. https://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/

3. https://www.rabbitmq.com/queues.html#runtime-characteristics

4. https://www.rabbitmq.com/ha.html#master-migration-data-locality

5. https://github.com/rabbitmq/rabbitmq-sharding

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael G]

Hi Karl and Mikel,

thanks for your help.

[Grigory Starinkin]

Hi,

I assume this is a question about exchange performance (route lookup).

Performance depends on exchange type.

Fanout exchanges are the fastest in terms of route lookup.

Direct exchange has constant lookup complexity, if I remember correctly. 

You can find more about complexity of topic exchanges here:

https://www.erlang-solutions.com/blog/rabbit-s-anatomy-understanding-topic-exchanges.html

https://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/

http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/

Thank you,

Grigory Starinkin 

On 21 Mar 2019, at 14:43, Michael G <michae...@gmail.com> wrote:

Hi Karl and Mikel,

thanks for your help.

--