Low performance with C# RabbitMq.Client package compared to Python or Java consumers

[Marco Zuiderwijk]

We have a sender application in the East-US region. This one sends messages to a RabbitMq instance which is also hosted in the East-US region. We use CloudAmqp for our RabbitMq instances. The messages are consumed from an application which is running in the Western Europe instance. I created 3 different versions of this application: 1 written in C# (using RabbitMQ.Client 6.6.0), 1 written in Python (using pika 1.3.2) and 1 written in Java (using amqp-client 5.16.0). There is no logic to process the message in the consumer applications. The only thing that is done is count the number of received messages and  each second display the number of messages. The python consumer processes > 500 messages / sec (and sometimes > 900), the Java consumer > 300 messages / sec. But my C# consumer only processes 12 messages / sec. This is too slow to process all the messages in the queue. For every consumer I started my implementation with the example code from the RabbitMQ site for the "Hello world" example. The message size in this test is 80kb. Can someone help me to understand why the performance of the C# consumer is so bad compared to the other consumers?

[Marco Zuiderwijk]

Some additional information. When I change the version of the RabbitMQ.Client to 5.2.0, I receive the messages in a higher rate which is comparable to the message rate received with the Java consumer.

[Luke Bakken]

Hello,

It's nearly impossible to assist you without seeing the test code you have written. Please provide code I can clone, compile, and run via a git repository.

We can either continue the discussion via google groups, or in the .NET client repository, here: https://github.com/rabbitmq/rabbitmq-dotnet-client/discussions

Team RabbitMQ knows that the performance of the .NET client is comparable to the Java one, so there must be something in your code or environment.

Thanks,

Luke

[Marco Zuiderwijk]

Hello Luke,

See the attachments for the source code of my Java and C# consumers

Kind regards,

Marco

[Luke Bakken]

Thanks Marco,

I will test this by publishing a batch of messages, then running your consumer code. This will take the publishing side out of it.

What version of .NET are you using?

[Marco Zuiderwijk]

I am using .net 7. 

In this test I was using different regions for the sender, queue and consumer. I also tested this with all the components in one region. This improves the message rate, but is not the way I want to have it working.

[Luke Bakken]

Hi Marco,

I took your code and put it into runnable form here, including a Makefile - https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY

In the future, it would be appreciated if the extra time to set up runnable projects were taken, because setting them up myself took about 30 minutes. This may not seem like much, but Team RabbitMQ is small and provides significant free software and free support, so anything that can make diagnosis easier is appreciated.

Having said that, I can't reproduce your issue. I am running the consumer code using a Ubuntu 22.04 virtual machine (WSL2 on Windows 11), and RabbitMQ 3.12 is running on a separate workstation (Arch Linux, Erlang 26). The systems are connected using 1Gb ethernet.

I have attached transcripts of running the publisher and consumers to the repository itself:

https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY/blob/main/csharp-transcript-shostakovich.txt

https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY/blob/main/java-transcript-shostakovich.txt

Have a nice weekend -

Luke

[Marco Zuiderwijk]

Hello Luke,

I appreciate the work you are doing to help me, and I am sorry for the extra work you had to do to make my examples work. I don't use these groups a lot, so I wasn't aware of this. The next time I share some code I will take care of this.

To be sure we tested the same setup, I have the following questions:

• Can you tell me which regions you used for the test? When I connect the consumer to a RabbitMQ instance in the same region, the message rate is much higher compared to a connection from Western Europe to a RabbitMQ instance in East-US.
• With which message size have you been testing? I experienced the message rate of 12 messages per second when using messages of 80kb. With smaller messages, the message rate will be much higher. 

The C#, Python and Java consumers I created are used from the same virtual machine. So there is no difference on the infrastructure which can explain the difference in message rates. The logic in these consumers is also almost the same as you could have seen in the code I sent you.

In the meantime I also tested my C# consumer with an older RabbitMQ.Client version (5.2.0). To my surprise with this version I was able to consume almost the same amount of messages per second as the Java implementation. After I upgraded this to package version 6.0.0 the message rate was at 12 messages / second again. I made no other changes in my consumer application. Is there anything changed in the implementation which can cause such a big difference in received messages?

Kind regards,

Marco

[Luke Bakken]

Hi Marco,

I detailed my test environment in my response. I am not using a cloud environment.

I will re-run my tests with 80Kb messages when I have a chance.

How different is the test code you provided compared your real consumer application?

Can you re-test with the following code running on your C# application startup?

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/a0321c6c2203cedeacdd5c1ec5bcdc924e85a842/projects/TestApplications/CreateChannel/Program.cs#L20

Thanks,

Luke

[Luke Bakken]

Hi Marco,

In my local test environment performance is still similar between the Java and .NET code using 80KiB messages.

Thanks,

Luke

[Marco Zuiderwijk]

Hello Luke,

I used the program you sent me to test the connection. Because I saw a difference on the received message rates between RabbitMq.Client version 5.2.0 and version 6.6.0, I also tested with both versions. The test application was running in Western Europe and it connects to a RabbitMQ instance in East-US.

RabbitMQ.Client 5.2.0
Took 823980.5237 ms
Took 801695.5125 ms

RabbitMQ.Client 6.6.0
Took 817612.7372 ms
Took 812592.7311 ms

For this test the average times are almost the same. 

You are using your local connection to do your tests. I think it is essential to connect to a RabbitMq instance in another region to experience the same issues like I had. We are also working together with a team from AWS. They experienced the same difference in message rates when they compared the C# and the Java client libraries. I also did a test with the C# and Java consumer running on the same machine at the same time. So, the infrastructure and CPU and memory usage are the same for both consumers. In this test the Java consumer receives > 300 messages per second, and the C# consumer receives only 12 messages per second.

Kind regards,

Marco

[Luke Bakken]

Hi Marco,

If the .NET client had performance issues such as these for everyone, believe me, we would hear about it. Everything so far points to either the AWS infrastructure (any load balancers in there?), or the VM running the .NET client code in your environment. I can't imagine that there is an issue with the .NET runtime installation because I have never heard of such a thing but I guess it's possible.

You yourself demonstrated that performance is equivalent in one of your tests that uses AWS.

Could you please answer the following questions that I asked earlier?

• How different is the test code you provided compared your real consumer application?
• Can you re-test with the following code running on your C# application startup? https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/a0321c6c2203cedeacdd5c1ec5bcdc924e85a842/projects/TestApplications/CreateChannel/Program.cs#L20

I don't have the time to set up an AWS environment to provide free support.

I recommend taking a TCP traffic capture of the .NET client's interaction with RabbitMQ. Do so in an environment that exhibits similar performance to the Java client, and one that has the performance issues. Then, compare. You may see differences in timings, TCP window sizes, that sort of thing. If you would like to provide the two captures to me, I can also take a look.

Thanks,

Luke

On Tuesday, November 7, 2023 at 5:15:30 AM UTC-8 marco.zu...@beyondsports.nl wrote:

Hello Luke,

[Michal Kuratczyk]

I'm fairly sure it's related to the TCP buffer sizes. This would explain why you need large messages

to trigger the problem and why there's no problem on localhost (TCP buffers are much larger

on the localhost by default).

https://www.rabbitmq.com/networking.html#tuning-for-throughput-tcp-buffers

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/469d5bc9-c4a8-4210-915e-4fbaef9213een%40googlegroups.com.

--

Michał

RabbitMQ team

[Marco Zuiderwijk]

Hello,

For me it is also impossible to believe that this issue is caused by something in the RabbitMQ.Client itself. I agree that a lot more users would have experienced these issues when this was the case. I am looking for some directions where I can search for a possible solution.

Because you suggested to look at the TCP traffic, I installed WireShark. You can see a fragment of the test results in the attachments. As you can see in the filenames, I compared a run using version 6.6.0 of the RabbitMQ.Client package and another run using version 5.2.0. I also tested this with a Java consumer, these results were comparable to the results of the run with version 5.2.0 of the RabbitMQ.Client. The biggest difference is the PSH flag which is set on a lot of the messages when using version 6.6.0. This is not set when using the older version. Can you explain this based on the changes between both RabbitMq.Client packages? Or is this related to the TCP buffer size like mentioned by Michal?

Kind regards,

Marco

On Thursday, 9 November 2023 at 08:47:08 UTC+1 SALMINI SHEMBAZI wrote:

On Thu, Nov 2, 2023, 3:18 PM Marco Zuiderwijk <marco.zu...@beyondsports.nl> wrote:

We have a sender application in the East-US region. This one sends messages to a RabbitMq instance which is also hosted in the East-US region. We use CloudAmqp for our RabbitMq instances. The messages are consumed from an application which is running in the Western Europe instance. I created 3 different versions of this application: 1 written in C# (using RabbitMQ.Client 6.6.0), 1 written in Python (using pika 1.3.2) and 1 written in Java (using amqp-client 5.16.0). There is no logic to process the message in the consumer applications. The only thing that is done is count the number of received messages and  each second display the number of messages. The python consumer processes > 500 messages / sec (and sometimes > 900), the Java consumer > 300 messages / sec. But my C# consumer only processes 12 messages / sec. This is too slow to process all the messages in the queue. For every consumer I started my implementation with the example code from the RabbitMQ site for the "Hello world" example. The message size in this test is 80kb. Can someone help me to understand why the performance of the C# consumer is so bad compared to the other consumers?

DISCLAIMER:

The information contained in this message or any of its attachments may be confidential and is intended for the exclusive use of the addressee(s). Any disclosure, reproduction, distribution or other dissemination or use of this communication is strictly prohibited without the express permission of the sender. The views expressed in this email are those of the individual and not necessarily those of Beyond Sports B.V. or affiliated companies. This email and any response may be monitored by Beyond Sports B.V. and Sony Security Operations Center to be in compliance with Sony's global policies and standards.

---

Beyond Sports B.V.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/77a8e3c7-df34-42a1-a463-04c1636aba0fn%40googlegroups.com.

[Karl Nilsson]

I'm wondering if the frame max negotiated by the client is different between the two versions. Can you compare what you see on the connection page for the two different versions in your

environment?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/a4617b8a-3637-4ad7-8aff-6610722101cfn%40googlegroups.com.

--

Karl Nilsson

[Marco Zuiderwijk]

The frame max is the same in both situations. I added  some screenshots of both situations. The only difference I see is in the data rate to the client.

Kind regards,

Marco Zuiderwijk

[Marco Zuiderwijk]

We use CloudAmqp for our RabbitMQ environment. I don't have access to configuration files and cannot change the TCP buffer size in one of the settings. Besides that, I still don't understand why I can have a higher message rate with a Java, Python or a C# consumer created wiht RabbitMQ.Client 5.2.0 compared to a C# consumer created with RabbitMQ.Client version 6.6.0 with the same settings for the TCP buffer size.

Kind regards,

Marco

[Luke Bakken]

Hi Marco,

I'm still waiting on a response to this question and request:

• How different is the test code you provided compared your real consumer application?
• Can you re-test with the following code running on your C# application startup? https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/a0321c6c2203cedeacdd5c1ec5bcdc924e85a842/projects/TestApplications/CreateChannel/Program.cs#L20

My understanding is that you have set up an AWS environment with your C# client application, using version 6.6.0 of the RabbitMQ.Client library, that does perform the same as your Java and Python test applications (in this message: https://groups.google.com/g/rabbitmq-users/c/9_ohuUbX9NY/m/z27Pvm6dAwAJ).

If that is true, you must compare this environment with the problematic one(s) to figure out every possible difference.

Thanks,

Luke

[Marco Zuiderwijk]

Hello Luke,

• The test code is a very simplistic implementation compared to our real consumer application. However, the connection, channel and consumption is configured the same as in our real consumer application
• I have send the results of this test on the 7th November. I tested with my setup across different regions and with RabbitMQ.Client version 5.2.0 and RabbitMQ.Client version 6.6.0. No big differences between both implementations here:

• 
  RabbitMQ.Client 5.2.0
  Took 823980.5237 ms
  Took 801695.5125 ms
  
  RabbitMQ.Client 6.6.0
  Took 817612.7372 ms
  Took 812592.7311 ms
  

I am sorry that this is not clear. But I don't have a situation where the version 6.6.0 of the RabbitMQ.Client version is performing the same as the Java, Python or RabbitMQ.Client version 5.2.0. I installed all consumer implementations on the same setup. So, the environment and configuration is the same for every consumer. The consumer created with RabbitMQ.Client version 6.6.0 is always having a low message rate compared to the other consumers. This setup is created on Azure instead of AWS. Although I also tested my consumer on a EC2 instance, which was giving the same results.

The only difference I found until now is in the TCP communication (shown in the message from November 9th: https://groups.google.com/g/rabbitmq-users/c/9_ohuUbX9NY/m/h3BeRdCbBAAJ). When using the RabbitMQ.Client version 6.6.0 a lot of messages have a PSH flag set which is not set in the communication when using RabbitMQ.Client version 5.2.0. But I don't have any experience with TCP communication at all, so I don't know what this means. And maybe this is also not the place for this question?

Thanks for you help,

Marco Zuiderwijk

[Luke Bakken]

• I have send the results of this test on the 7th November. I tested with my setup across different regions and with RabbitMQ.Client version 5.2.0 and RabbitMQ.Client version 6.6.0. No big differences between both implementations here:

• 
  RabbitMQ.Client 5.2.0
  Took 823980.5237 ms
  Took 801695.5125 ms
  
  RabbitMQ.Client 6.6.0
  Took 817612.7372 ms
  Took 812592.7311 ms
  

I am sorry that this is not clear. But I don't have a situation where the version 6.6.0 of the RabbitMQ.Client version is performing the same as the Java, Python or RabbitMQ.Client version 5.2.0.

This is confusing.

You send two sets of numbers showing similar performance between .NET client 5.2 and 6.6, then you say you don't have a situation where 6.6 performs the same as 5.2.

Which is it? 

[Marco Zuiderwijk]

Hello Luke,

The results of the 7th November test is with the sample application you send me. This application is creating channels and that has the same performance with both RabbitMQ.Client libraries. The difference between this application and mine is that I am receiving messages. I experience the performance differences when receiving messages from the RabbitMQ instance.

Kind regards,

Marco Zuiderwijk

[Luke Bakken]

Please open a pull request to my test code to make it like your test code.

https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY

In addition, make the change I requested here, re-run your tests in your environment, and report back. I have asked this several times:

• Can you re-test with the following code running on your C# application startup? https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/a0321c6c2203cedeacdd5c1ec5bcdc924e85a842/projects/TestApplications/CreateChannel/Program.cs#L20

[Marco Zuiderwijk]

Sorry, but I don't know what you want me to change. You shared a test to create a lot of channels. Do you want me to create all these channels before I start receiving the data? Of do I have to register a EventingBasicConsumer on all the chanels? Can you please explain a little bit more?

[Luke Bakken]

That link points to this line of code;

ThreadPool.SetMinThreads(16 * Environment.ProcessorCount, 16 * Environment.ProcessorCount);

Please add it to your application's startup and re-run your tests.

[Marco Zuiderwijk]

Hello Luke,

Adding this line doesn't change my results. I also tried to use asynchronous processing of the messages. However the results are still the same, see the attachement. The value for the arguments to the SetMinThreads() method is 64 in my tests.

I tried to create a PR to your test code with the latest version of my consumer application. But this resulted in a 403 when I tried to push my code to the repo. I included the changed program.cs in this message. All other files are still the same.

Kind regards,

Marco Zuiderwijk

[Luke Bakken]

Hi Marco,

Thanks for reporting the results. In order to open a PR, you must fork my repository first:

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests

[Luke Bakken]

Thanks for the code Marco.

In my environment, performance is still the same - https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY/blob/main/doc/2023-11-15/transcript-0552PST.txt

Can you reproduce this issue every single time in your environment?

[Marco Zuiderwijk]

Hello Luke,

We can reproduce this issue every time we do these tests.

Thanks to my colleague Joey Bleeker we might have a solution for this issue. As stated before, we don't have this issue when using version 5.2.0 of the RabbitMQ.Client. He focused on the changes between those versions and found this commit: https://github.com/rabbitmq/rabbitmq-dotnet-client/commit/ff627be358269985da306ca0e76a0e7a962b4366. In this commit a fixed value is added for the ReceiveBufferSize and SendBufferSize when creating a Socket in method DefaultSocketFactory() of ConnectionFactoryBase. It looks like the TCP feature Window Scaling is not being applied after this change.

I created a fork of the latest RabbitMQ.Client repository and did some tests while investigating the setup of the connection with WireShark. You can see the results of these tests in the attachments. With RabbitMQ.Client version 5.2.0 (TCP_5_2_0.png) you can see on the last line the window size of 263424. With RabbitMQ.Client version 6.6.0 (TCP_6_6_0.png) this is 65536 (from the ReceiveBufferSize). The forked version of the RabbitMQ.Client where setting the buffer size is removed (TCP_fork.png) shows the same results as the RabbitMQ.Client version 5.2.0. I created PR Removed ReceiveBufferSize and SendBufferSize to improve message rates by MarcoZuiderwijkBS · Pull Request #1414 · rabbitmq/rabbitmq-dotnet-client (github.com).

Kind regards,

Marco Zuiderwijk

[Luke Bakken]

Excellent! Thank you VERY much. I will review this change to make sure it doesn't negatively affect performance in other environments.

⭐⭐⭐⭐⭐⭐⭐

[Luke Bakken]

Hi Marco,

Thanks again for your investigation. You can work around the current behavior of setting those buffer sizes by creating the ConnectionFactory with a custom SocketFactory func:

ConnectionFactory cf = new()
{
    SocketFactory = (AddressFamily af) =>
    {
        var socket = new Socket(af, SocketType.Stream, ProtocolType.Tcp)
        {
            NoDelay = false
        };
        return new TcpClientAdapter(socket);
    }
};

[Luke Bakken]

Hmm, TcpClientAdapter isn't public, so you'll have to copy the code to implement your own ITcpClient. Yuck. I will improve this in version 7 of the client.

[Luke Bakken]

Here is the workaround implemented in the test project:

https://github.com/lukebakken/rabbitmq-users-9_ohuUbX9NY/blob/main/cs/Program.cs

[Luke Bakken]

Hi Marco,

Please give this pre-release a test:

https://www.nuget.org/packages/RabbitMQ.Client/6.7.0-rc.1

[Marco Zuiderwijk]

Hello Luke,

Thank you for this new release candidate. I used it in my consumer and it is working like a charm!

From you previous messages I got the impression that you wanted to make a change related to the SocketFactory and / or TcpClientAdapter. The new rc only contained the removal of the 2 buffer sizes. This will affect the performance improvement why the original change was made (as was mentioned by Michael Klishin in a comment on my PR). Is this something that is still an issue, or will this be solved in version 7?

One small comment on the unit test you added: you expect a default buffer size of 8192 bytes. During our tests, we saw that the default buffer size is 65536 bytes. There seems to be a difference between the documentation and the current implementation.

Kind regards,

Marco

[Luke Bakken]

Hi again Marco -

 

From you previous messages I got the impression that you wanted to make a change related to the SocketFactory and / or TcpClientAdapter. The new rc only contained the removal of the 2 buffer sizes. This will affect the performance improvement why the original change was made (as was mentioned by Michael Klishin in a comment on my PR). Is this something that is still an issue, or will this be solved in version 7?

I don't think the original change had as great an effect as everyone thought. I will provide an example of how to re-implement your own buffer sizes via the SocketFactory func.

 

One small comment on the unit test you added: you expect a default buffer size of 8192 bytes. During our tests, we saw that the default buffer size is 65536 bytes. There seems to be a difference between the documentation and the current implementation.

Thanks for noticing that. In my testing environment, when buffer sizes are set to 1024 they actually end up being about 2000 bytes, which is why I don't check for an exact value. I'll come up with a different strategy.

Expect an official 6.7.0 release today or tomorrow.

Thanks,

Luke 

[Luke Bakken]

Hi Marco,

One more pre-release before version 6.7.0 - https://www.nuget.org/packages/RabbitMQ.Client/6.7.0-rc.2

Would you mind dropping it into your environment? I have made the TcpClientAdapter class public, thus changing the public API of the project. Just want to make sure it's binary compatible with existing applications. It should be but a check would be great.

Thanks,

Luke

[Marco Zuiderwijk]

Hello Luke,

Thanks again for the new release candidate. 

I did the following tests:

- Build and run a consumer application using the new release candidate

- Copy the new RabbitMQ.Client.dll into an existing build and run the consumer

Both tests were successful. So, no issues with the compatibility between the different release candidates.

I am very happy that we found the cause of our performance issue. We appreciate your work and the speed of creating a new RabbitMQ.Client release. We will wait with updating our software until an official release is created.

Kind regards,

Marco Zuiderwijk

[Luke Bakken]

Thanks again Marco!

I have pushed the official 6.7.0 release and will announce it on this mailing list today -

https://www.nuget.org/packages/RabbitMQ.Client/6.7.0

Have a great weekend

Luke

[Marco Zuiderwijk]

Hello Luke,

Thanks again for your quick response. I integrated the new version into our software.

Kind regards,

Marco Zuiderwijk

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Luke Bakken <luker...@gmail.com>
Sent: Friday, November 17, 2023 4:58 PM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] Re: Low performance with C# RabbitMq.Client package compared to Python or Java consumers

 

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/9_ohuUbX9NY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/508bf2a5-88c7-4b85-b247-0c7dc36c5bd4n%40googlegroups.com.