RabbitMQ .Net Performance

[matth...@matthew1471.co.uk]

Reposted from https://github.com/rabbitmq/rabbitmq-dotnet-client/issues/97 as deemed to be not an "Issue", grateful for any comments or help :-) :

----------------------------------------------
Hi,
I've been working on an application that handles a LOT of messages and after optimising my code to every bit of its life believe I have found a performance issue in rabbitmq-dotnet-client v3.5.3.

Visual Studio 2013 performance analysis points out that the majority of the program now spends its time executing this:

```c#
                    while (m_running)
                    {
```
projects\client\RabbitMQ.Client\src\client\impl\Connection.cs

I've noticed that if I have 1 channel open, then I am unable to keep up with the volume of messages, however if I open another channel (presumably MainLoopIteration then runs for longer and has more things to do) say about 3, then the application performs perfectly and spends more time reading messages.

The ironic thing is just running 3 instances of exactly the same channel code against the same connection is enough to improve the performance enough to keep up in real time.

For my consumers I use ManualResetEvent and AutoResetEvent as that is generally now preferable in threads than using very tight while loops like while (true) or while (m_running) and I wondered if the RabbitMQ client needed the same?

I'm new to the RabbitMQ .NET Client source, does someone more familiar with the code base know if the while loop can be replaced with a semaphore like that used by ManualResetEvent to further improve its performance? Presumably the frame would be written to a BlockingCollection and then the loop would wait for the signal?

See : http://stackoverflow.com/questions/8881396/cpu-usage-increasing-up-to-100-in-infinite-loop-in-thread and http://stackoverflow.com/questions/7402146/cpu-friendly-infinite-loop

Thank you for your time,
Matthew
----------------------------------------------

[Michael Klishin]

On 17 June 2015 at 11:07:31, matth...@matthew1471.co.uk (matth...@matthew1471.co.uk) wrote:
> Reposted from https://github.com/rabbitmq/rabbitmq-dotnet-client/issues/97
> as deemed to be not an "Issue", grateful for any comments or help
> :-) :

From the same thread.

Opening new channels does not "slow down" I/O loop and even if it did, I don't see how your consumer throughput may increase from that. Using more channels and consumers means a higher degree of concurrency in the client.

In 3.5 you can provide your own task scheduler and dispatch is concurrent, with per-channel ordering preserved. Previously all operations were dispatched on the I/O thread. That was an obvious
bottleneck. 

I don't know what kind of profiler you used but sampling profilers will always indicate that the I/O loop is the top item even though all it does is reading and parsing incoming data, potentially waiting for I/O. This cannot be parallelised in practice for a single connection.

The I/O loop can be migrated to newer kernel I/O API now that we require a recent .NET version but
for clients it won't make a huge difference. Same with attempts to optimise protocol parser.

Looking at what takes time and/or can be parallelised in your consumers has a lot more potential. Using more channels or even connections (if your link is very good, e.g. 100 GBit/s) is also more likely to yield noticeable improvements.

And then there is a blog post [1] about consumers and QoS settings' effect on throughput.

1. https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[matth...@matthew1471.co.uk]

>will always indicate that the I/O loop is the top item .. potentially waiting for I/O

when using wait handles / semaphores the Visual Studio 2013 Performance Analysis doesn't show up conditional checks in my threads any more (which now use BlockingCollections and ManualResetEvents instead of while loops).. **I think it's more that VS is seeing the _conditional check_ as being executed an excessive number of times rather than the actual I/O**.. the I/O does come up in the list of top items (which is like you said, exactly what should be expected) but the while loop conditional check should not.

I don't seem to have a problem with the I/O functions, purely the _conditional check_. It's running the conditional check a lot (even when there aren't any frames available).

I'll take a look at those blog posts on my commute to work, thank you for your time and comments so far :-)

Regards,
Matthew

[Michael Klishin]

On 17 June 2015 at 11:35:12, matth...@matthew1471.co.uk (matth...@matthew1471.co.uk) wrote:
> I don't seem to have a problem with the I/O functions, purely
> the _conditional check_. It's running the conditional check
> a lot (even when there aren't any frames available).

It is needed to make the I/O thread terminate when we detect a heartbeat timeout,
or another unrecoverable issue. Implementing heartbeats in terms of socket timeouts
doesn't work out for various reasons, we are moving away from it.

This check always happens in a thread that does nothing but I/O and protocol
method dispatch. I doubt that has a significant impact on your system's
throughput ;)

But yes, there are definitely ways to make the I/O part more efficient by using
a different I/O API.