Single vs Multiple Exchanges

[Vilius Šumskas]

Hello,

 

I‘m looking for a general advice when to use single exchange or when consider splitting into multiple exchanges.

 

The use case is pretty simple.

Service A (publisher) receives events from REST endpoint and publishes them to RabbitMQ.

Service B (publisher and consumer) listens for messages from Service A, enriches them with additional data and published back to RabbitMQ.

Service C (consumer) listens for messages from Service B and sends them to data analytics storage.

 

There will be ~1000 event messages per second coming in from Service A. Under ideal conditions the processing of messages in all three “cycles” would happen in seconds. However we design a system for 24 hour persistence using dead letter queues and TTL, in case part of the system goes down.

 

Previously we would just simply use one topic exchange for all producers. All consumers would have their dedicated queues and binding keys. But now I’m doubting if that was the right call. I cannot explain it, but it doesn’t feels right. Even though all services are essentially one domain the messages before and after data enrichment are quit different (much larger). We also don’t need wildcard support and I’ve read that topic exchanges are slower than direct. Not sure if that’s still the case for RabbitMQ 3.13/4.x series, as most of the blog posts about exchanges I found are quite old.

 

Even if we switch from topic to direct exchange there is still a question should we split into two exchanges or not?

 

I’m also concerned about how publishing from two different service to the same exchange affects performance. If let’s say suddenly Service A has to publish a large amount of messages, will it also slow down Service B publisher? In other words, are exchanges single threaded in RabbitMQ?

 

--

    Best Regards,

    Vilius

 

[Luiz Carlos Faria]

With HTTP, requests go to endpoints and, after processing, the results are persisted in database tables.

In a similar way, in RabbitMQ an exchange plays the role of the “endpoint” that receives messages, while a queue is roughly analogous to a table—the place where messages are held for consumers.

It’s not a perfect one-to-one, but it’s a useful mental model.

----

Publishers know Exchanges, Routing Keys and Headers.

Consumers know Queues.

----

Slightly different purposes can use the same exchange, with different routing data.

Huge different purposes demand different exchange.

I hope this can help you!

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB81340272B43E49B6448A0F769207A%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

 ​ ​ ​ ​    ​ 

[Vilius Šumskas]

OK. So what are the main factors deciding on when messages purposes needs separate exchanges?

 

--

    Vilius

 ​ ​ ​ ​    ​ 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CAOByxJLnL8vYkVG2X-3TTptjAHhy5aAzF%2BzvbRpYfD7wQjU7Ng%40mail.gmail.com.

[Michal Kuratczyk]

The act of routing a message is a small fraction of the overall publishing time.

With ~1000 messages per second, I'd be surprised if you notice any difference,

so this is mostly a theoretical discussion. 

Having said that, it sounds like the default exchange might be all you need:

https://www.rabbitmq.com/docs/exchanges#default-exchange

Direct exchanges will always be faster than topic exchanges, simply because they are much simpler:

a direct exchange is a lookup table that maps routing keys to destinations, like this:

key1 -> queue1

key2 -> queue2a

key2 -> queue2b

When you publish a message, RabbitMQ (specifically the channel process if you use AMQP 0.9.1) will query

this table, get all the values (usually just one, but in this example a lookup for "key2" would return two results)

and send the message to all of the destinations.

The default exchange is even faster/simpler than other direct exchanges, because no actual lookup is performed:

the routing key is the name of the destination and that's it.

A topic exchange needs to support wildcards, and therefore uses a more complex lookup algorithm

to handle partial matches. However, if you only have a handful of bindings, it hardly

matters how complex the lookup process is. If you had many thousands of different bindings on a topic

exchange, then you would start seeing a larger overhead.

Multiple exchanges can be required or useful in some situations, but it's often more about project

organization / responsibilities than strictly technical requirements. If you have a single RabbitMQ

instance handling different message flows, it can be useful to clearly separate exchanges related

to different flows and likely managed by different people and/or applications. Performance

can be a reason but only if you have a lot of bindings and a high throughput. 

You can easily experiment with all of this using https://perftest.rabbitmq.com/. A quick test on my machine:

# declare a quorum queue "qq" and publish to it through a topic exchange with 1 binding:

$ perf-test -e amq.topic -k "foo.bar" -t topic -qq -u qq -c 1000 -y 0

...

sending rate avg: 57758 msg/s

If I create more bindings on the topic exchange, the performance degrades:

# declare 10000 topic bindings for the same queue

for i in $(seq 10000); do rabbitmqadmin declare binding --source amq.topic --destination qq --destination-type queue --routing-key foo.bar.$
i; done

# publish again (same as above)

$ perf-test -e amq.topic -k "foo.bar" -t topic -qq -u qq -c 1000 -y 0

...

sending rate avg: 42942 msg/s

Let's use the default exchange for comparison 

# publish to the previously declared qq through the default exchange:

perf-test -p -e "" -k qq -c 1000 -C 1000000 -y 0

...

sending rate avg: 63837 msg/s

So in this test, the default exchange is 10% faster than a topic exchange with 1 binding

and 50% faster than a topic exchange with 10000 bindings. However, you can't just

assume it's always the case - if I create 10k bindings with the key "bar.bar.$i" (instead

of the "foo.bar.$i" created above), there's no drop in performance at all (I still get ~58k/s,

just like I do with 1 binding), because perf-test publishes with "foo.bar" key and the topic

matching algorithm we use can quickly discard all bindings that start with "bar".

So the performance of topic exchanges depends not only on the number of bindings

but also on what they are (data distribution).

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB81346C6739A70415E502099B9207A%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Vilius Šumskas]

Thank you for detailed explanation Michal.

 

One more question regarding publishing to the exchange. Is receiving a message, scanning through binding, keys and routing to appropriate queue, a single threaded operation on RabbitMQ side?

 

--

    Vilius

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWL5%3D91uH5L6wM7h%3DYfwVy2hnZLEPgji%2BHeHGb0DpWfxfg%40mail.gmail.com.

[Michal Kuratczyk]

No. An exchange is not a process[1], it's a dataset. The routing happens in the process that receives the messages from the publisher, for example:

* in AMQP 0.9.1, it's the channel process: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_channel.erl#L1224

* in AMQP 1.0, it's the session process: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_amqp_session.erl#L2505

* in MQTT, it's the MQTT processor process: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_mqtt/src/rabbit_mqtt_processor.erl#L1651

So in each case, the act of routing is performed by a process that is dedicated to a specific connection.

I'm using a vague meaning of "connection" here, but the point is it's not a single process for all publishers.

[1] some less popular exchange types have processes associated with them, but I'm focusing on the main exchange types here

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB813470E21C5C0B3A51D2E7679206A%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Vilius Šumskas]

OK. It is more clear now.

 

One additional question. Given that exchange is a dataset, not a process, does it have a size limit and is there a way to monitor the size (in case I deploy new producer but consumers are not yet deployed and no queues or bindings exist)? I haven’t found anything in RabbitMQ Management UI or Prometheus metric list.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWJDQ6KKbcfFLeLnAXZS6ZKyx%2BdZFjCdnP0ghVv%3DQCc-uw%40mail.gmail.com.

[Michal Kuratczyk]

There are no enforced limits. Technically you are limited by the amount of memory available, but I'm yet to hear

about anyone who needs so many bindings. There are certainly systems with hundreds of thousands of bindings

on a single exchange.

You can use the management API to list/count the bindings:

curl -s -u guest:guest 'http://localhost:15672/api/exchanges/%2F/amq.direct/bindings/source' | jq length
3

More info: https://rawcdn.githack.com/rabbitmq/rabbitmq-server/main/deps/rabbitmq_management/priv/www/api/index.html

But I'm not sure why you want to count bindings in your "publishers are present before consumers/queues" example.

You can either just publish and the messages will be dropped or you can publish with the "mandatory" flag

and the publisher will receive a negative confirmation if there are no matching bindings.

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB8134EB9741044A841EBD0DA29200A%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Vilius Šumskas]

I don‘t really want to count bindings. Since exchange is a dataset, I had an impression that exchange, without any queues and bindings attached to it, just stores the messages indefinitely until at least one queue and binding appears. By size, I meant the sum of all those messages. From your answer it looks like messages in such case are just dropped, so I now understand that it is unnecessary.

 

The reason I was asking is that when microservices are developed, to have a proper separation, you usually define the destination (exchange) and the routing key in the producer, but the queue and the binding key is defined in the consumer. For various cases, this could result in exchanges not having queues attached to them at least temporarily.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWKnzQfqPbEU%2BQw-U6KHvf2ww5nfJpdRS%3DBTDHag5ECKDg%40mail.gmail.com.

[Michal Kuratczyk]

An exchange is a dataset of bindings, not messages. Messages are never "inside an exchange"

(again, there are some plugins like the delayed exchange plugin where the answer is more

nuanced, but let's focus on direct/topic/fanout/header exchange types here). Taking a direct

exchange as an example, it's really just a lookup table like:

routing_key1 -> destination1

routing_key2 -> destination2a

routing_key2 -> destination2b

That's the dataset I was talking about. When you publish a message "to an exchange" it just means that your channel

(assuming AMQP0.9.1) will take the routing key you provided, look up that key in the exchange table and forward

the message to all the destinations (usually 1 for a direct exchange). If there are no matching bindings, the message

is either dropped (default) or "returned" (when the mandatory flag is set).

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB8134B2CF4F6F4D90CB8E28CC9200A%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Vilius Šumskas]

Thank you again Michal.

 

Such deep knowledge how RabbitMQ internals work really helps to understand it now.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWLE%3D1C6CUL3vmrK-KEMWPsfp7066KJoOORo2EcBYtnKAQ%40mail.gmail.com.