RMQ Crashing after : Slogan: eheap_alloc: Cannot allocate memory

[Samir G]

Hi,

These days I am seeing following error in RMQ very frequently. And its causing to crash RMQ.

Here is snippet of crash log.

=erl_crash_dump:0.2
Mon Aug 17 18:02:05 2015
Slogan: eheap_alloc: Cannot allocate 60647069488 bytes of memory (of type "old_heap").
System version: Erlang R16B01 (erts-5.10.2) [source] [64-bit] [smp:24:24] [async-threads:30] [kernel-poll:true]
Compiled: Wed Aug  7 13:42:03 2013
Taints: crypto,asn1rt_nif
Atoms: 26615
=memory
total: 68837813040
processes: 67926083424
processes_used: 67924319376
system: 911729616
atom: 793505
atom_used: 782179


Also I am seeing following errors from RMQ logs.

=INFO REPORT==== 17-Aug-2015::17:57:58 ===
vm_memory_high_watermark set. Memory used:71556727352 allowed:40540871065

=WARNING REPORT==== 17-Aug-2015::17:57:58 ===
memory resource limit alarm set on node 'r...@mq1001.corp'.

******************************

****************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

We are currently running

RabbitMQ v3.3.
Erlang vR16B01


Has anyone encountered similar issue? Do you know what might be the root cause.


Thanks!
--SamirG

[Michael Klishin]

This is a runtime problem. Upgrade to Erlang 17.x or 18.0.

MK

[Samir G]

Hi Michael,

I have been running Erlang vR16B01 for more than a year and never encountered this issue before. Is there any reason why it will cause RMQ to crash?

Also is Erlang 17.x or 18.x compatible with RabbitMQ v3.3?


Thanks!
--Samir

[Robert Raschke]

The crash you are seeing is because Erlang failed to allocate 60GB of memory in one go. Seems fairly reasonable to stop at that point.

One of the more common reasons for having such a huge allocation request comes from error logging of processes with very large state.

Is it possible that you have some queues with a lot of messages, really a lot? If Rabbit MQ has a failure of some sorts on a queue, I believe it logs a lot of information about the queue.

HTH,
Robby

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

You can use those versions with 3.3.x.

MK

[christ...@triokasoftware.com]

Hi Robby, is there any way to avoid such failures. Am currently facing the same issue my queue is getting filled with a huge load and that is expected, am thinking of way to handle this failure gracefully.

[Michael Klishin]

Please start new threads for new questions.

[1][2][3] all seem to be relevant here, too.

1. http://www.rabbitmq.com/memory-use.html

2. http://www.rabbitmq.com/lazy-queues.html

3. http://www.rabbitmq.com/maxlength.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ