After upgrade from rabbitmq 3.2.0 to 3.5.0 on MacBookPro, erlang crashes upon stopping rabbit

[vitaly numenta]

Running on MacBookPro i7 with 16GB RAM and SSD drive with the latest Mac OS X Yosemite. Until the last hour or so, I was running RabbitMQ 3.2.0. There were no erlang crash logs in /usr/local/var/log/rabbitmq/. I start/stop rabbitmq pretty regularly. After upgrading to RabbitMQ 3.5.0, I noticed that there would be a fresh erl_crash.dump file in /usr/local/var/log/rabbitmq/ each and every time that I shut down rabbitmq. 

I uploaded the last erl crash dump to this github gist: https://gist.github.com/vitaly-krugl/105ec1a0e316889af800.

I start rabbitmq as follows: rabbitmq-server -detached

And stop it like this: rabbitmqctl stop

Here is the tail of my log:

=INFO REPORT==== 3-Apr-2015::00:47:51 === 

Server startup complete; 10 plugins started.

 * rabbitmq_management_visualiser

 * rabbitmq_management

 * rabbitmq_web_dispatch

 * webmachine

 * mochiweb

 * rabbitmq_mqtt

 * rabbitmq_stomp

 * rabbitmq_management_agent

 * rabbitmq_amqp1_0

 * amqp_client

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

Stopping RabbitMQ

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

stopped MQTT TCP Listener on [::]:1883

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

stopped STOMP TCP Listener on [::]:61613

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

stopped TCP Listener on 127.0.0.1:5672

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

Stopped RabbitMQ application

=INFO REPORT==== 3-Apr-2015::00:50:26 === 

Halting Erlang VM

Here is my rabbitmq status info with all the versions:

$ rabbitmqctl status

Status of node rabbit@localhost ...

[{pid,1297},

 {running_applications,

     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.5.0"},

      {rabbitmq_management,"RabbitMQ Management Console","3.5.0"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.0"},

      {webmachine,"webmachine","1.10.3-rmq3.5.0-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.0-git680dba8"},

      {rabbitmq_mqtt,"RabbitMQ MQTT Adapter","3.5.0"},

      {rabbitmq_stomp,"Embedded Rabbit Stomp Adapter","3.5.0"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.0"},

      {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.5.0"},

      {rabbit,"RabbitMQ","3.5.0"},

      {os_mon,"CPO  CXC 138 46","2.2.15"},

      {inets,"INETS  CXC 138 49","5.10.2"},

      {mnesia,"MNESIA  CXC 138 12","4.12.1"},

      {amqp_client,"RabbitMQ AMQP Client","3.5.0"},

      {xmerl,"XML parser","1.3.7"},

      {sasl,"SASL  CXC 138 11","2.4"},

      {stdlib,"ERTS  CXC 138 10","2.1"},

      {kernel,"ERTS  CXC 138 10","3.0.1"}]},

 {os,{unix,darwin}},

 {erlang_version,

     "Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},

 {memory,

     [{total,46482480},

      {connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,5616},

      {queue_procs,237048},

      {queue_slave_procs,0},

      {plugins,650616},

      {other_proc,14288832},

      {mnesia,77328},

      {mgmt_db,154712},

      {msg_index,1838568},

      {other_ets,1416328},

      {binary,66808},

      {code,20974593},

      {atom,744345},

      {other_system,6027686}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},

      {amqp,5672,"127.0.0.1"},

      {stomp,61613,"::"},

      {mqtt,1883,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,6259844710},

 {disk_free_limit,50000000},

 {disk_free,420516560896},

 {file_descriptors,

     [{total_limit,2460},

      {total_used,9},

      {sockets_limit,2212},

      {sockets_used,3}]},

 {processes,[{limit,1048576},{used,209}]},

 {run_queue,0},

 {uptime,4}] 

[vitaly numenta]

... and in case it matters, I upgraded RabbitMQ via:

brew update

brew upgrade rabbitmq

[Michael Klishin]

On 3 April 2015 at 10:57:37, vitaly numenta (vitaly.kru...@gmail.com) wrote:
> Here is the tail of my log:

That log suggests RabbitMQ server has stopped successfully. I suspect that Homebrew packaging
does something that makes rabbitmqctl exit with an error (it starts an Erlang process as well). 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[vitaly numenta]

I installed 3.2.0 via brew, too, but never had an erl crash dump. What do you recommend as a recovery strategy, as it seems somewhat unsettling to have this perfectly reproducible crash in Erlang each and every RabbitMQ shutdown.

Many thanks!

Vitaly

[Jean-Sébastien Pédron]

Hi!

What if you try the standalone package for Mac OS X?
http://www.rabbitmq.com/install-standalone-mac.html

Do you see the same crash?

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Michael Klishin]

On 3 April 2015 at 11:16:34, Jean-Sébastien Pédron (jean-se...@rabbitmq.com) wrote:
> What if you try the standalone package for Mac OS X?
> http://www.rabbitmq.com/install-standalone-mac.html

That's what Homebrew uses. 

[Michael Klishin]

On 3 April 2015 at 11:16:34, Jean-Sébastien Pédron (jean-se...@rabbitmq.com) wrote:

> What do you recommend as a recovery strategy, as it seems somewhat
> unsettling to have this perfectly reproducible crash in Erlang
> each and every RabbitMQ shutdown.

Vitaly,

erlang_crash.dump often does not suggest a RabbitMQ or Erlang runtime bug. It can can be a misconfiguration,
an Erlang path problem, a SIGUSR1 signal sent to the process, there may be no file descriptors left for
VM to use on boot, etc. All of which can be packaging-induced.
Yes, Homebrew packaging probably has changed in the 18 months between 3.2.0 and 3.5.0.

I cannot reproduce this issue with a generic UNIX package on OS X (3.5.0) but see an erl_crash.dump
with a standalone OS X build of 3.5.1 (will be announced today). RabbitMQ successfully starts and stops,
though .

Feel free to use our generic UNIX package but the issue seems to be a harmless package-specific thing.

[Michael Klishin]

 On 3 April 2015 at 12:07:02, Michael Klishin (mkli...@pivotal.io) wrote:
> I cannot reproduce this issue with a generic UNIX package on
> OS X (3.5.0) but see an erl_crash.dump
> with a standalone OS X build of 3.5.1 (will be announced today).
> RabbitMQ successfully starts and stops,
> though .

Looking at the dump suggests the VM terminates very early in the process, long before RabbitMQ attempts to start
("init terminating in do_boot ()" and there are only 3 Erlang processes running).

Almost certainly it's just something silly that starts `erl` with incorrect parameters.

[Alvaro Videla]

We have opened a bug for this: https://github.com/rabbitmq/rabbitmq-server/issues/96

Thanks for the initial report

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vitaly numenta]

Thank you for looking into it. What does it mean for the rest of us who are using 3.5.0 or 3.5.1? What are the performance and stability implications of this erl crash?

Thank you,

Vitaly

[Alvaro Videla]

If epmd is running (which should be running when you start RabbitMQ) then you shouldn't be affected by this problem.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

None. And it is specific to the OS X standalone build.