RabbitMQ TLS performance hit

[Alexander B]

Hi, 

we are testing RabbitMQ TLS performance.

Configuration: Centos 7, Erlang 19.3 + Rabbit 3.6.8 with inter-node TLS enabled. Server is cloud instance 2VCPU + 8G RAM.

On a node we have 1 tcp listener and 1 TLS one.

we use Java Perf Tool for testing. Publishing with 1 publisher, 1 consumer on topic exchange  (HA queue with 1 slave)

TCP: 6K msg/s

TLS: 2K msg/s

so we have 3X performance hit.

We tried:

Checked that TLS sessions are reused and TLS negotiation is not taking place all the time

Tried to specify different ciphers for Rabbit to use

Checked that EVP encryption hardware acceleration is enabled on the instance.

Installing different distributions of erlang (esl-erlang)

Using HiPE to compile rabbit

Non of the above gave a significant performance boost.

From other people's experience, is it normal for TLS to be that of a performance problem?

What other things we could try?

Thank you

[Michael Klishin]

You haven't provided any code for your tests but yes, TLS has

overhead and it is particularly visible for workloads with short lived connections.

I believe Mirantis developers reported somewhat similar numbers in the past.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander B]

Hi Michael,

thank you. I use java performance tool: https://www.rabbitmq.com/java-tools.html

I execute 

bin/runjava com.rabbitmq.perf.PerfTest -h amqps://_username:_password@ip  --type topic  --queue test-2 --size 1000 --producers 1 --consumers 1 --autoack true

[Michael Klishin]

OK, I see you use PerfTest which opens a long running connection.

Besides RabbitMQ runtime having to do more work with TLS so does the client.

Try comparing CPU utilisation, I bet it's also quite different.

On 30 Mar 2017, at 02:38, Alexander B <alex...@gmail.com> wrote:

--

[Michael Klishin]

I was suggesting comparing CPU utilisation with and without the TLS.

On Thu, 30 Mar 2017 at 04:18, Alexander B <alex...@gmail.com> wrote:

The CPU utilisation on server is 95-100%

On the client - 12-15%

On Thursday, March 30, 2017 at 2:00:35 PM UTC+13, Michael Klishin wrote:

OK, I see you use PerfTest which opens a long running connection.

Besides RabbitMQ runtime having to do more work with TLS so does the client.

Try comparing CPU utilisation, I bet it's also quite different.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[Alexander B]

Another interesting thing we noticed - no different between TLS performance of Erlang 18.3 vs Erlang 19.3, 

while we expected one - 19.3 contains newer version of erlang crypto that supports EVP - hardware encryption acceleration. Our CPUs support it as well, but no difference can be observed.

On Thursday, March 30, 2017 at 2:18:41 PM UTC+13, Alexander B wrote:

The CPU utilisation on server is 95-100%

On the client - 12-15%

On Thursday, March 30, 2017 at 2:00:35 PM UTC+13, Michael Klishin wrote:

[Alexander B]

The CPU utilisation on server 

with TLS - 100% on 2k msg/s
no TLS - 40% on 3K msg/s

So yes, CPU takes the hit

[Alexey Lebedeff]

Hi Alexander,

3x is the exact drop we've observed in our performance tests, when both
AMQP and inter-node TLS is enabled. And you can't do anything on
Erlang/RabbitMQ side to fix this.

Just consider this worst case scenario for a message roundtrip in a
cluster:
1) Message is published via AMQP
2) A node where client connection resides pushes this message to a node
where a queue process resides
3) The queue process delivers the message to yet another node, where a
consumer connection is open
4) Message is delivered to the consumer using AMQP

So when TLS is enabled everywhere, each of these steps include
encryption and decryption of the whole message.

My personal opinion is that you should use some external mechanism for
securing inter-node communication - something like IPSec. And use TLS
only for AMQP.

But in our production environments we've decided not to use TLS at all =)

Best,
Alexey