publish to all queues

[revooru nischal]

Hi ,

   I am using rabbitmq in my IOT architecture. Devices connect using mqtt architecture and my server is connected using amqp using spring cloud stream. 

My exchange is a topic exchange which is suitable for mqtt. And I have multiple queues for each device where I send the routing key as device ID with which the device is subcribed on. 

This works fine, but I have few usecases where i need to send to all queues like a broadcast message.

Is this possible ? 

if yes what should be routing key which needs to be sent ?

[Wesley Peng]

revooru nischal wrote:
> This works fine, but I have few usecases where i need to send to all
> queues like a broadcast message.

fanout exchange?

[Wesley Peng]

For instance, I have this consumer script:

$ cat consume.rb
require "bunny"

conn = Bunny.new
conn.start
ch = conn.create_channel

exchange = ch.fanout("fanout")
queue = ch.queue("").bind(exchange)

queue.subscribe(:block => true) do |delivery_info, metadata, payload|
puts "Received #{payload}"
end

And this producer script:

$ cat publish.rb
require "bunny"

conn = Bunny.new
conn.start

ch = conn.create_channel
exchange = ch.fanout("fanout")

10.times do
data = rand.to_s
exchange.publish(data)
end



I open two terminals, run each consumer.rb in each terminal separately.

Then open the third terminal, run publish.rb.

And we will see two consumers get and print the same message items.

Consumer1 output:

$ ruby consume.rb
Received 0.8439900790883322
Received 0.7934571880375225
Received 0.7594074915448258
Received 0.6846564266710206
Received 0.04857228429430649
Received 0.5275599997437828
Received 0.9342227328889011
Received 0.3994398464282922
Received 0.2484340114543926
Received 0.6501834053032844

Consumer2 output:

$ ruby consume.rb
Received 0.8439900790883322
Received 0.7934571880375225
Received 0.7594074915448258
Received 0.6846564266710206
Received 0.04857228429430649
Received 0.5275599997437828
Received 0.9342227328889011
Received 0.3994398464282922
Received 0.2484340114543926
Received 0.6501834053032844


This is what the fanout exchange works like.
Thanks.

[revooru nischal]

Hi ,

   Ya I have read about fanout exchange. But mqtt works on topic exchange right ?

And fanout exchange publishes messages to all queues every-time. But I want it to be done only on specific condition on topic exchange

[Wesley Peng]

revooru nischal wrote:
> And fanout exchange publishes messages to all queues every-time. But I
> want it to be done only on specific condition on topic exchange

you can have given number queues bound to topic exchange with specific
routing key.

for example, you have 3 queues bound to exchange with routing key
"test_a", and 2 queues bound to exchange with routing key "test_b".

Thus messages with routing key "test_a" will be delivered into all 3
queues, and messages with routing key "test_b" will be delivered into
all another 2 queues.

Does this work for you?

[revooru nischal]

I think this works fine. its good.

But i am only worried about scalability. If the message needs to be published to each and every queue it creates a copy for each of it. 

So this needs to be used only when very much required. 

[Wesley Peng]

revooru nischal wrote:
> But i am only worried about scalability. If the message needs to be
> published to each and every queue it creates a copy for each of it.

For high throughput rabbitmq optimization you could check this article:
https://www.cloudamqp.com/blog/2018-01-08-part2-rabbitmq-best-practice-for-high-performance.html

Thanks.

[revooru nischal]

thanks

[Michael Klishin]

Your understanding is correct: a mass fanout places a copy into every queue, so until 3.9 comes out with stream queues (which offer repeatable reads), this workload is best used sparingly or avoided.

[revooru nischal]

Hi Michael,

   Thats great thanks. Can I know some approximate release date of 3.9

[Michael Klishin]

We do not make any ETAs promises on releases.