Memory Leak in .NET Client 3.5.0?

[Mike Simpson]

Hi,

I just spent quite a bit of time debugging a memory leak in one of our apps, an ASP.NET Web Api site that sends messages to RabbitMQ via the C# client.  I had recently updated to 3.5.0, noticed increasing memory with little/no usage, and updated to 3.5.1 as well with no improvement.  As a ballpark measure the site's memory increased from 200 MB to 2 GB in a couple of days with nobody hitting it.

I profiled the app and saw that the RabbitMQ Connection class seems to be holding onto a lot of byte[] instances.  One leak may be caused by failure to dispose of instances of BinaryReader created within NetworkBinaryReader... these are created for one-time use on the fly, in ReadSingle() and ReadDouble(), and never disposed.  I built my own version of the client with 'using' blocks to fix that, and I saw some improvement, but it didn't completely fix the problem.

I believe I also saw that I had a lot of instances of Connection, which was concerning because we try to hold onto a single Connection instance for the lifetime of the app by bootstrapping it as a singleton in our IoC container.  But we have a wrapper around RabbitMQ.Client and if the connection goes down, the wrapper reconnects, so that might have been a symptom of the connection dropping a lot.

Interestingly, we have a bunch of other services using the same wrapper around the same RabbitMQ.Client version, and as far as I can tell, they haven't experienced the memory leak.  I don't know what's different about the one site, but the others are mostly Windows services (i.e. installed and run as daemons).

Rolling back to 3.4.3 fixed the memory leak.  Anybody else see the same behavior?

Mike

[Michael Klishin]

Try nightly builds. Timers used by heartbeats implementation in 3.5.x were not disposed.

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Simpson]

Thanks for the quick reply!  I'll see about doing that, but we use the Nuget package so if I switch to a nightly build I'll need to embed it in our wrapper package.  I may wait for the next package release.

Cheers,

Mike

[jimx...@gmail.com]

I got the same memory leak using RabbitMQ.Client.dll 3.5.3. At beginning, I am using EasyNetQ too, I thought something is wrong with EasyNetQ, after removing EasyNetQ, I created my own wrapper and got the same memory leak issue too under a unstable internet connection, the memory increased very quickly to 4G. It causes serious problem.

Hi Mike, does the issue fixed in your application?

I will try to use the old version 3.4.3.

Thanks,

Jim

[Michael Klishin]

On 16 June 2015 at 22:57:28, jimx...@gmail.com (jimx...@gmail.com) wrote:
> I thought something is wrong with EasyNetQ, after removing
> EasyNetQ, I created my own wrapper and got the same memory leak
> issue too under a unstable internet connection, the memory increased
> very quickly to 4G.

4 GB is a lot of RAM. Please profile to see what takes it. It would take quite
a high connection connection churn to leak that many timer objects!

Using the 3.4.4 client with a server version other than 3.4.x is perfectly  fine.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Mike Simpson]

I haven’t tried anything newer than 3.5.0.  We rolled back to 3.4.3 and are still using that.

 

One alternate approach might be to turn off heartbeats in RabbitMQ.Client, and instead add your own error handling and connection recovery logic.  I used to have some logic like that in our library*, but I ripped it out recently.

Mike

 

*I’m planning to open-source it at some point, as soon as I can remove the company-specific pieces.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/aaRuh7llRGM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

 On 16 June 2015 at 23:02:38, Mike Simpson (mike.s...@ignitionone.com) wrote:
> One alternate approach might be to turn off heartbeats in RabbitMQ.Client,
> and instead add your own error handling and connection recovery
> logic.

Note that heartbeat is only a way of detecting peer unavailability sooner, and
not required (or necessarily involved in) connection recovery. One can both disable
heartbeats and use automatic connection recovery.

[jimx...@gmail.com]

Yes, I spent a lot of time to get around of the memory leak issue. 

Yes, I did profile using VS20013, the top 3 object types are Hashtable (FactoryRecord), RabbitMQ.Client.Framing.Impl.Connection and BufferedStream. Please see the memory profile screen shots, posted in the link https://groups.google.com/forum/#!topic/easynetq/hMZRwtCi4os

[Michael Klishin]

On 16 June 2015 at 23:58:35, jimx...@gmail.com (jimx...@gmail.com) wrote:
> Yes, I did profile using VS20013, the top 3 object types are Hashtable
> (FactoryRecord), RabbitMQ.Client.Framing.Impl.Connection
> and BufferedStream.

This suggests your system is leaking connections. Do you close them with Connection#Close or #Abort?

Do you use automatic recovery?

[jimx...@gmail.com]

The memory usage is normal if the internet connection is good.

I tried both handle the connection manually  and automatic recovery, both got the same result of huge memory usage. The following is the CloseConn code.

public void CloseConn()

        {

            if (ModelConsume != null)

            {

                try

                {

                    ModelConsume.Close(200, "Goodbye");

                    ModelConsume = null;

                }

                catch { }

            }

            if (ModelPublish != null)

            {

                try

                {

                    ModelPublish.Close(200, "Goodbye");

                    ModelPublish = null;

                }

                catch { }

            }

            if (rmqConnection != null)

            {

                try

                {

                    rmqConnection.Close();

                    rmqConnection = null;

                }

                catch { }

[Michael Klishin]

 On 17 June 2015 at 00:46:22, jimx...@gmail.com (jimx...@gmail.com) wrote:
> The memory usage is normal if the internet connection is good.
>
> I tried both handle the connection manually and automatic recovery,
> both got the same result of huge memory usage.

If you don't use automatic recovery, then RabbitMQ client will _never_ open new connections
if there's a connectivity issue. Which means your own code opens them and is thus responsible
for closing them.

[Frank Thomsen]

Hi,

I have the exact same problem with version 3.5.4 of the client library, even though the release notes states that the problem is fixed.

Using a memory profiler I can see that a lot of byte[]s and strings are not released. I have just reverted to version 3.4.3 in production which fixes the problem, but this is not optimal since newer releases fixes other bugs. So I look forward to new version.

I am willing to share code examples but not on a public channel.

./frank

[Michael Klishin]

Can you profile where they originate or produce a small example that demonstrates this?

MK