[rabbitmq][rabbitmq-sharding]Speed regression between Debian RabbitMQ 3.3.5 and Official 3.6.1

[bapt...@databerries.com]

Hi,

I was testing several config in order to vertical scale RabbitMQ for our usage. We have around 500 bytes persistent messages and we do not want any message to be consumed more than once.
I wanted to test the sharding-plugin and since it is included in 3.6.1, my approach was to test it both with the 3.3.5 version provided by Debian Jessie and 3.6.1 debian package provided by RabbiMQ's website both with Erlang 17.3 provided by Debian.

I used the PerfTest tool from the java client (version 3.6.1) and a 8 core high CPU machine from Google Cloud  for RabbitMQ

I first ran several commands like below to test several queues running in parallel from a distant machine:

/runjava.sh com.rabbitmq.examples.PerfTest -c 1000 -f persistent --uri amqp://user:passwd@host:5672

The result were the following:

For RabbitMQ 3.3 we have the following results:

• 1 queue: 8,900 msg/sec load: 3.80
• 2 queues: 16,000 msg/sec load: 5.20
• 3 queues: 21,000 msg/sec load: 7.14
• 4 queues: 26,000 msg/sec load: 7.24 (max speed on instance)

For RabbitMQ 3.6.1:

• 1 queue: 6,000 msg/sec load: 1.23
• 2 queues: 10,300 msg/sec load: 2,60
• 3 queues: 13,000 msg/sec load: 4.90
• 4 queues: 16,000 msg/sec load: 5.60
• 5 queues: 18,000 msg/sec load: 6.90
• 6 queues: 19,000 msg/sec load: 7.51
• 7 queues: 20,300 msg/sec load: 7.80
• 8 queues: 20,700 msg/sec load: 7.76 (max speed on instance)

The measurement are measured by eye and measure the average value after a few minutes of running. While the measurement are not precise the difference between the two versions of RabbitMQ is significant enough. While each queue seems to be more efficient in load there is an important performance regression.

After that I tried rabbitmq-sharding performance using the configuration from this discussion.

I created a shard attached manually a consumer to each queue with command such as:

./runjava.sh com.rabbitmq.examples.PerfTest -p -e myshard -u "sharding: myshard - rabbit@rabbitmq-test - 0" -c 1000 -f persistent -x0 -y1

--uri amqp://user:passwd@host:5672

and then run:

./runjava.sh com.rabbitmq.examples.PerfTest -p -e myshard -c 1000 -f persistent -x1 -y0 -K

--uri amqp://user:passwd@host:567

For RabbitMQ 3.3.5 with sharding:

• 1 producer:
  • 2 queues: 18,600 msg/sec load: 3.13
  • 4 queues: 16,900 msg/sec load: 3.45
  • 6 queues: 15,000 msg/sec load: 3.63
• 2 producers:
  • 2 queues: 33,800 msg/sec load: 5.93
  • 4 queues: 27,800 msg/sec load: 5.09
  • 6 queues: 25,500 msg/sec load: 6.00
• 4 producers:
  • 2 queues: 40,000 msg/sec load: 6.28
  • 4 queues: 46,000 msg/sec load: 8.00
  • 6 queues: 45,600 msg/sec load: 8.00

For RabbitMQ 3.6.1 with sharding:

• 1 producer:
  • 2 queues: 9,600 msg/sec load: 2.63
  • 4 queues: 11,800 msg/sec load: 4.20
  • 6 queues: 13,000 msg/sec load: 7.30
  • 8 queues: 13,400 msg/sec load: 9,60
• 2 producers:
  • 2 queues: 9,500 msg/sec load: 2.40
  • 4 queues: 12,500 msg/sec load: 6.00
  • 6 queues: 13,700 msg/sec load: 6.70
  • 8 queues: 15.600 msg/sec load: 9.40
• 4 producers:
  • 4 queues: 12,500 msg/sec load: 5.31
  • 6 queues: 14,000 msg/sec load: 7.70
  • 8 queues: 15,800 msg/sec load: 9.60
• 6 producers:
  • 6 queues: 14,400 msg/sec load: 7.08
  • 8 queues: 15,800 msg/sec load: 9.70
• 8 producers:
  • 6 queues: 14,000 msg/sec load: 7.08
  • 8 queues: 15,800 msg/sec load: 9.36

The results are even worse with sharding if we compare 3.6.1 and 3.3.5.
I was surprised since I was expecting the opposite. Does anyone has an idea why there is such a speed regression (in msg/sec) ?

Thanks,

Baptiste

[dfed...@pivotal.io]

Hi,

Can you provide broker logs for this tests?

[Baptiste Desbiolles]

Here are the broker logs. I did the tests on April 12th.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Sbt_up3Poig/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dfed...@pivotal.io]

I cannot find 3.6.1 logs there. Can you provide them as well?

[Baptiste Desbiolles]

I don't have them anymore. I will run the tests again.

[Michael Klishin]

There are many releases between 3.3.5 and 3.6.1:

http://www.rabbitmq.com/changelog.html.

Please try finding a release where the results

start being meaningful different first.

On Mon, Apr 18, 2016 at 2:47 PM, Baptiste Desbiolles <bapt...@databerries.com> wrote:

I don't have them anymore. I will run the tests again.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ