RabbitMQ windows service says it's started when it isn't ready

[Mat Burton]

Hey all,

A few years ago I posted this: https://groups.google.com/forum/#!topic/rabbitmq-discuss/hx8pDobWixo

Back then all we could to do was to set our services which depend on RabbitMQ to use delayed auto start.

However it's still causing us problems as the RabbitMQ broker somewhat regularly takes more than 3 minutes to start.

Given time has passed, is there now any way to correct RabbitMQ's service behavior such that it only enters the 'running' state once it's actually ready?

If not how do other people work around RabbitMQs behavior with respect with machine restarts and dependent apps on windows?

Many thanks,

Mat B

[Mat Burton]

It looks like I'm not the only one who's been hitting this problem: https://github.com/rabbitmq/rabbitmq-server/issues/1551

Does anyone know if this a limitation of RabbitMQ or of Erlang's erlsrv?

[Luke Bakken]

Hi Mat,

This is an Erlang / erlsrv limitation -

https://github.com/erlang/otp/blob/OTP-21.2.3/erts/etc/win32/erlsrv/erlsrv_service.c#L767-L777

You can see that the code starts an Erlang VM, sets the status to pending and then running immediately after.

You can use WMI or some other mechanism to get the PID of the erl.exe process that is running RabbitMQ, then use the rabbitmqctl wait --pid PID command to wait for RabbitMQ to fully start.

Thanks,

Luke

[Mat Burton]

Hey Luke,

May thanks for the reply and a plausible workaround.

[Michael Klishin]

I should mention that in 3.7.11 we have a new command, `rabbitmq-diagnostics await_startup [--timeout N]` which

does not require a pid file or pid but does require CLI tool connectivity.

Perhaps it would allow you to reduce the number of moving parts in your deployment? We would appreciate some feedback

specifically from Windows users :)

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Mat Burton]

Thanks Michael,

In our instance I think making all our service apps which depend on RabbitMQ somehow consume that command line tool would be somewhat painful and wouldn't fix all of the problems caused by the improper service behaviour.

In the end we implemented a similar workaround, we changed all our services to constantly try to connect to RabbitMQ for 10 minutes while requesting more start-up time.

We chose this workaround as it doesn't require any further configuration or dependencies and it's easy to revert if Erlang is improved in the future.

It does however add a fair amount of code complexity and has some unwanted side effects.

The code before was simple as:

• We could assume that RabbitMQ was available (this assumption was wrong though due to the Erlang limitation)
• We didn't have to periodically request any further time for start-up from windows, the default 30s is plenty
• If we failed to connect during start-up we'd fail fast and could report that the configuration or app was borked

One side effect is that, if an app or configuration is regressed in certain ways, our deployment system won't fail fast and will just give a time-out and we'll be much less certain about what was wrong.

Another side effect is that our sites and APIs (HTTP apps rather than headless apps) will give error responses until RabbitMQ starts rather than the 503s they should respond with.

In practice all of these apps are load balanced but some error responses are still likley to slip through after a restart and will cause alerts.

The real danger is that the error responses could cause the node to be removed from the load balancer soon after a restart.

I think the only real fix is for Erlang to support a way for RabbitMQ to report when it's finished starting.

Does any of the above help?

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

It's the deployment tools that are meant to use that command, not apps.

[Mat Burton]

Hey Michael,

In that case that command line tool wouldn't really help much, it's machine restarts that are by far the biggest problem.

Upgrading RabbitMQ would be another example where there is no deployment.

Machine restarts are much more common though due to updates.

[Michael Klishin]

RabbitMQ does report its status to systemd which has a mechanism for that. If there is a similar mechanism on Windows

we'd definitely like to know but as you correctly pointed out, what applications think of a node's state and what the

service manager thinks of its state are different problems.

This gets even more fun in clusters where any client can connect and perform operations on any node.

The best idea I keep arriving at is that nodes should enable listeners (that accept client connections) as their last step.

WDYT?

[Michael Klishin]

I include "upgrades" into "deployment scenarios" in my mind but I see what you are saying. See

my earlier response, application detection of service availability is not what I was thinking about

when I mentioned `await_startup`.

Kubernetes provides a readiness check command for pods, we basically need a similar thing but to be used by applications.

[Michael Klishin]

I filed an issue for consideration that's somewhat relevant here [1]. Thank you for sharing the details

and how you adapted your system to cope.

1. https://github.com/rabbitmq/rabbitmq-server/issues/1869

[Mat Burton]

Thanks Michael,

If my understanding is correct the similar mechanism in windows is to tell the SCM (through win32) when the service has finished starting.

When the SCM starts a service it runs the executable and sets the state to "Starting".

It's then up to the application to tell the SCM when it's fully operational (the "Running" state).

These states allow the SCM to start/stop dependent services sensibly, it knows the dependency chains.

The problem isn't that the RabbitMQ's service doesn't do that, the problem is that erlsrv does that immediately even though it hasn't even begun to execute the app yet.

Hence there's not much RabbitMQ itself can do, erlsrv has already told the lie ;-)

Is there any scope to change erlsrv?

[Michael Klishin]

We can contribute to Erlang/OTP as it is open source software with pretty reasonable maintainers. I don't know when

we'd get to doing so, though.

But thank you for the explanation, at least I understand the limitations in erlsrv now.

[Michael Klishin]

3.7.14 will have an update in this area [1]. It's only a small step forward

and is not Windows-specific, though.

1. https://github.com/rabbitmq/rabbitmq-server/issues/1869

Michael Klishin:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mat Burton]

Hey Michael,

Sorry for the slow reply.

That's awesome :-) many thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/ctLroWGmqGQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.