RabbitMQ restarting randomly on Windows

[biiiipy]

We have been running 3 node RabbitMQ 3.6.2 (Erlang 18.2.1) cluster for about a year and we have seen random node reboots. 

We started testing RabbitMQ 3.7.2 (Erlang 20.2) which has the same issue. There is no crash dump, no errors, nothing, just entry in the log:

(the first line likely is not related to this, it's kind of a known bug for epmd on windows to crash)

2018-01-11 15:41:58.241 [warning] <0.368.0> epmd does not know us, re-registering rabbit at port 25672  

2018-01-11 15:46:04.545 [info] <0.73.0> RabbitMQ is asked to stop...
2018-01-11 15:46:05.670 [info] <0.73.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_tracing
    rabbitmq_shovel_management
    rabbitmq_management
    rabbitmq_amqp1_0
    rabbitmq_shovel
    amqp_client
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbitmq_message_timestamp
    rabbitmq_auth_mechanism_ssl
    rabbitmq_delayed_message_exchange
    rabbit
    mnesia
    rabbit_common
    os_mon
    amqp10_client
    amqp10_common
2018-01-11 15:46:05.670 [info] <0.73.0> Stopping application 'rabbitmq_tracing'

 

...and the rest of shutdown log is as usual and it ends with:

2018-01-11 15:46:05.748 [info] <0.73.0> Halting Erlang VM with the following applications:
    ranch_proxy_protocol
    ranch
    ssl
    public_key
    asn1
    crypto
    inets
    recon
    jsx
    xmerl
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel

 How could we diagnose the issue?

[Luke Bakken]

Hello,

What version of Windows? Does this happen at the same time every time? Have you checked the system event logs?

I am assuming you installed RabbitMQ using the .exe installer and are thus running RabbitMQ as a Windows service.

Thanks -

Luke

[Michael Klishin]

Historically one very common case where we've seen nodes being "unknown" to epmd is when the user starts

RabbitMQ under their own account and then signs out. This is exactly the scenario that led us to adding the "re-registration" feature.

Can something like this be going on? The node logs that it was told to shut down, are there any Windows service management logs

you can consult?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[biiiipy]

Windows Server 2012R2 and Windows Server 2016. Windows Event logs has nothing

We have a custom setup - basically we bundled together Erland and RabbitMQ and configured with environment variables via .bat file. Windows service is installed using rabbitmq-service.bat. It could be that it's related to our custom setup, but that's not very likely, because it can work for days and maybe even weeks without a problem.

Michael, yes, we are running service on a server under a special service user, which isn't logged in as a typical user. But I'm not sure that this is related to RabbitMQ restarting.

Thanks for fast responses!

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

The "is asked to stop" message comes from this line:

https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit.erl#L458-L470

As far as I know, this will only happen when something like the service controller or rabbitmqctl actually sends RabbitMQ a "stop" message. Earlier I asked this question: "Does this happen at the same time every time?" - is this the case?

[biiiipy]

It happens randomly. I will try to gather more information, maybe catch a crash as it happens.

[Luke Bakken]

Just to be clear, this isn't a crash or unexpected event. Something is initiating a clean shutdown of RabbitMQ.