System.OutOfMemoryException

[Peter Tern]

Hi,

We have a lot of services consuming messages using subscribe. These have been running in production for a year now. The services are created using C# and RabbitMQ client.

The load has been at most 100 000 messages per hour but now we are doing a initial load and the total amount of messages is 1 200 000 messages. Then we get this:

2018-02-23 09:33:10.0460 TUI.Transformers.Facit.Price.Data.FacitPriceData

An unexpected error occurred when trying to save Facit Price Message.

Error message=Exception of type 'System.OutOfMemoryException' was thrown.

2018-02-23 09:33:10.0616 TUI.Transformers.Facit.Price.Service.Program

An unexpected exception occurred during attempt to send message to the error queue:

Already closed: The AMQP operation was interrupted:

AMQP close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0,

cause=System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.IO.BinaryReader.ReadBytes(Int32 count)

   at RabbitMQ.Client.Impl.Frame.ReadFrom(NetworkBinaryReader reader)

   at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()

   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()

   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop().

“

 

Where “TUI.Transformers.Facit.Price.Data.FacitPriceData” is the name of the service.

Hope that someone can help me out with this.

Regards,

Peter

[Luke Bakken]

Hi Peter,

What version of the .NET / C# client, RabbitMQ and Erlang are you using? What is your typical message size? How much memory is available on the machine running the .NET client code?

When you say "initial load", are you reading messages from disk or some other source and sending to RabbitMQ? Can you provide the code that is doing this operation?

A stack trace is never enough information to diagnose an issue.

Thanks,

Luke

[Michael Klishin]

This kind of messages usually comes down to two things:

 * This app consumes messages without a sensible prefetch limit or in automatic acknowledgement mode [1] without understanding the risk of doing so.

 * The app uses more threads than developers expected. This would result in the same error messages on the JVM but I'm not sure about .NET.

In both cases, please take a look at [1] and the relevant metrics in the management UI (on the queue and channel pages).

Beyond that, only a heap dump and profiler can tell what uses RAM in your application. It may or may not be related to RabbitMQ.

Don't guess, collect data instead.

1. http://www.rabbitmq.com/confirms.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ