fanout Performance RabbitMQ 3.5.7

[Fei Jiang]

I am working on rabbitmq performance. I am using https://github.com/openstack/oslo.messaging/blob/master/tools/simulator.py  measure the performance.

So, I am running the tool for following use-cases and test setup

Memory :  50GB

CPU:  40 Vcpu

RabbitMQ version : 3.5.7

Broker and client is in different machine.

1  test fanout exchange,  1024 queue bind to test exchange,  200 publisher, publish use basic.confirm, and consumer use  auto acknowledge, aslo use ha-mode  exactly(1+1);

Use-case: Non persistent publisher and non-durable subscriber;  Publish speed  only  20msg/s.

What is the reason behind this scenario. that publish speed onlys 20 msg/s, it too low.

What is publisher speed should be in this Use-case?

Thanks.

[Karl Nilsson]

Hi,

There are a few aspects of this test that can make things perform slowly. Firstly each publish needs to be copied to 1024 queues all of which need to perform replication. This does not come for free. Secondly to issue the publisher confirm the channel needs to gather responses from 1024 queues before it can do that for each message.

I'd suggest you try to relax some of your requirements to gain better performance. Remove ha policy for these queues or drop publisher confirms or alternatively re-design your system not to require a large fan-out topology.

Cheers

Karl 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Schröder]

2018-02-28 8:36 GMT+01:00 Fei Jiang <jiangf...@gmail.com>:
> RabbitMQ version : 3.5.7

Why do you test a version that's 26 months old when 3.7.3 exists?

Best
Martin

[Michael Klishin]

While that's a good point, the fundamental problem here is that a fanout to thousands queues is going to be expensive

anyway because each queue must get its own copy per protocol requirements. That's true for any version.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Fei Jiang]

Remove the ha policy for these queue， it can imporve prtformance greatly。

在 2018年2月28日星期三 UTC+8下午6:16:04，kjnilsson写道：

[Fei Jiang]

I delete the ha policy for these queues，  pulisher max speed 700 msg/s， improved greatly.

my machine have 64 core， RABBITMQ's CPU usage is no more than 1500%, what  is the reason. there is some rabbitmq config limit for it?

thank you.

[Michael Klishin]

The Erlang runtime by default will use all the cores it can detect. In the management UI you can go to the node

information page and see that number in the Advanced section.

With that many cores, scheduler-to-core binding strategy used becomes important. Also,

certain things in RabbitMQ are units of concurrency, most notably queues, connections and channels.

A single queue won't use 30 cores. 30 queues will. So check your workload as well.

Both topics were discussed multiple times before on this list. Search for "Erlang scheduler" and "concurrency", for example.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.