3.7.6 crashes periodically with Erlang slogan erl_child_setup closed

[Philipp Stratmann]

Hello,

RabbitMQ v3.7.6 periodically (like every 2h) crashes, writing an Erlang crash dump with slogan erl_child_setup closed. The full crash dump and rabbit log files are attached.

OTP version is 20.3.6. RabbitMQ is then automatically restarted and the clients (all Spring AMQP) reconnect until it happens again. The RabbitMQ log files shows no indication.

RabbitMQ version 3.7.5 is running fine.

System is a development system on a CentOS box with nearly no load. Configuration is default config.

It happens on an old system updated from 3.3.5 and on a new system with 3.7.6 being initially installed.And it's not being killed by the OOM killer.

Any hints how to fix this?

Cheers,

Philipp

[Luke Bakken]

Hi Philipp -

OOM is a common reason for this, and perhaps running out of file descriptors, though the log doesn't show that. I can see that the limit hasn't been raised on that system for RabbitMQ. The way that the log ends abruptly indicates an OOM event - could you please provide the output of dmesg or dmseg -t (if supported)?

Can you quantify "nearly no load"? What size messages, what rate, # of queues, that sort of thing.

Thanks,

Luke

[Philipp Stratmann]

Hi Luke,

I already checked OOM and it does not seem to be causing the issue. I nevertheless attached the output of dmesg -t

Load was 0.10/s published messages in the last hour, 22 exchanges, 119 queues. For more details see attached screenshot. Message size is less than 10 kB.

Cheers,

Philipp

[Luke Bakken]

Hi Philipp -

Thanks for that info. There is nothing obvious that stands out at this point. I would be interested in the contents of /var/log/messages or the message journal if you can provide it.

Another option would be for you to run this script to collect information that I can examine - https://github.com/rabbitmq/support-tools/blob/master/scripts/rabbitmq-collect-env

Thanks again -

Luke

[Philipp Stratmann]

Hi Luke,

please find the output from the script and /var/log/messages attached.

A little background. The machine is a VM running on virtual box and hosts the DB and RabbitMQ. Each developer has her own VM running locally. That's why the load is so low. We've been using the same box for the past two or three years without issues.

Our customers will be running RabbitMQ on either Linux or Windows. But as long as it's not running well on the dev system I can't recommend updating to the latest version.

Cheers,

Philipp

[Michael Klishin]

Hi Philipp,

The message you see in rsyslog means that the Erlang runtime had to terminate. There are absolutely

no errors, warnings or anything that looks suspicious to me in RabbitMQ logs.

See OS logs for messages from OOM killer events, SELinux messages and so on, and try a

different Erlang version, e.g. 20.3.8.1 or, if you can move to RabbitMQ 3.7.7, Erlang 21.0.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Actually right after I posted I found this:

Jul  5 13:40:20 PASX3x-DBVM rabbitmq-server: systemd unit for activation check: "rabbitmq-server.service"
Jul  5 13:40:20 PASX3x-DBVM systemd: Cannot find unit for notify message of PID 11593.
Jul  5 13:40:30 PASX3x-DBVM rabbitmq-server: Service still in 'activating' state, bailing out
Jul  5 13:40:30 PASX3x-DBVM rabbitmq-server: systemd READY notification failed, beware of timeouts
Jul  5 13:40:31 PASX3x-DBVM rabbitmq-server: completed with 3 plugins.
Jul  5 13:50:01 PASX3x-DBVM systemd: Starting Session 19 of user root.
Jul  5 13:50:01 PASX3x-DBVM systemd: Started Session 19 of user root.
…
Jul  5 14:40:15 PASX3x-DBVM systemd: rabbitmq-server.service operation timed out. Terminating.

So it's systemd that kills the (successfully started) node. It's not a common thing we see, what Linux distribution

is this on and how did you install Erlang and RabbitMQ?

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Philipp Stratmann]

Hi Michael,

[root@PASX3x-DBVM rabbitmq]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)

I used the rpm from bintray and installed via yum:

erlang-20.3.6-1.el7.centos.x86_64

rabbitmq-server-3.7.6-1.el7.noarch

Cheers,

Philipp

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Philipp Stratmann]

Hi Michael,

that looks a lot like https://github.com/rabbitmq/rabbitmq-server/issues/664 but that was fixed two years ago.

Anyway, seeing that 3.7.7 is out I gave it a try and after updating to 3.7.7 I haven't seen the issue yet. So, I'm good for now. It leaves a bad feeling, though.

Cheers,

Philipp

On Friday, July 6, 2018 at 11:49:02 AM UTC+2, Michael Klishin wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

There were no changes to RPM packaging in the last 2-3 patch releases that I can recall.

I don't know how more systemd logging can be enabled but if it happens again, please find a way to do that. RabbitMQ

has no idea why its runtime is terminated by systemd, so it won't log anything.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.