RabbitMQ Hangs Forever Waiting for PID File Creation

[Teresa Hume]

Hi,

I'm running into a problem getting rabbitmq up and running and am stumped as to what the issue is.

I'm using rabbitmq 3.6.6_1 on freeBSD 10.3-RELEASE.

RabbitMQ is included in rc.conf to start up on boot. It is running inside a freebsd jail, I've disabled ipv6 on the jail (removing ::1 from /etc/hosts as well as any ipv6 network interface definitions in /etc/rc.conf). At this point it is a fresh install, and I'm just trying to start it up using the default configuration.

Running `service rabbitmq start` hangs on the message: starting rabbitmq. I did some further digging into the rabbitmq startup script and found that it is forever waiting on the PID file to be created in /var/run/. The log directory for rabbitmq exists, but no logs are being created on trying to start up, so I can't see any indication of where the startup might be failing. I've also tried creating fake PID files for rabbitmq before launching it and the fake files are properly deleted, so I don't think it's a permissions issue in /var/run/.

Any ideas as to what might cause this issue? Or where I should be looking for additional logs/info on what might be failing?

Thanks!

[Michael Klishin]

See node logs. Their default locations are documented in http://rabbitmq.com/relocate.html.

They can vary between installation methods.

Tools such as strace/dtrace can help further.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[Teresa Hume]

Just a heads up for anyone who encounters this behaviour in the future.

It turns out the jail's hostname could not be resolved to an IP - we were able to determine this using `service rabbitmq status` as dtrace/strace were not an option on this machine and, like I mentioned in the original post, no rabbitmq logs were being generated at all (including node logs). The solution was to add the appropriate entry in /etc/hosts to link up the hostname with the jail's IP - apparently this can be an issue when freeBSD jails have their hostnames reset.

I'm not sure if this is the place to mention it, but this seems like a bug in rabbitMQs startup script. If it can't connect to the host after a certain period of time it should abort the connection and log relevant information instead of waiting forever and giving no indication of what might be wrong.

Thanks for the suggestions!

[Wedge Martin]

This is the most frustrating thing ever.  I was in a pet store the other day, saw a rabbit, and immediately wanted to stab it.   I have lost many hours of my life due to this crap. 

I love FreeBSD but what really pisses me off is the inability to use the service wrapper unless RabbitMQ is enabled in the rc.conf, and if it is enabled, you're guaranteed that your instance may never come back up again.

[Michael Klishin]

It would much easier for the maintainers and contributors to do something that improves things if instead of "this crap" and "I want to stab a rabbit"

we were provided with a set of steps to reproduce with a recent version (3.6.x is going to get another release or two before it EOL hits).

But it's not like open source users have any moral obligation to maintainers of the software they use in 2018, everyone is just entitled to getting everything for free,

including support and permanent updates, right?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Also worth mentioning that in 3.7.x, CLI commands that can support a timeout, do.

This includes `rabbitmqctl wait`. There was a couple of revisions to the nitty gritty details of

how it works when things fail driven by our own experience with RabbitMQ deployment automation.

So chances are things a somewhat different with 3.6.15 or 3.7.x.

Hostname resolution requirements are pretty visibly documented, too: http://www.rabbitmq.com/clustering.html#overview-hostname-requirements.

We will add a similar section to the CLI tools guide.

There are quite a few things that may need to resolve a hostname for one reason or another, some in CLI scripts and other in the actual apps,

and without a way to reproduce we can't know which one specifically is affecting the OP.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Wedge Martin]

Sorry Michael. I agree that I left out a fair amount of details. I am a big RabbitMQ fan and I lean on it and web stomp pretty heavily for a lot of things and for the most part it never lets me down, and I didn’t mean to imply that RabbitMQ itself was crap. It’s fantastic, the problem just seems to be the mix of FreeBSD and RabbitMQ ( and potentially whatever the hell I’m doing wrong. ). I apologize for my harsh tone.

Some details I left out:

- The bunny was an American Fuzzy Lop

- OS: FreeBSD 11.1-RELEASE-p8 (GENERIC) #0: Tue Mar 13 17:07:05 UTC 2018
rabbitmq-3.6.14_1
Name : rabbitmq
Version : 3.6.14_1
Installed on : Wed Mar 21 03:30:41 2018 UTC
Origin : net/rabbitmq
Architecture : FreeBSD:11:*
Prefix : /usr/local
Categories : net
Licenses : MPL11
Maintainer : olg...@FreeBSD.org
WWW : http://www.rabbitmq.com/
Comment : RabbitMQ is an implementation of AMQP
Options :
ADMIN : off
Annotations :
cpe : cpe:2.3:a:pivotal_software:rabbitmq:3.6.14:::::freebsd11:x64:1
repo_type : binary
repository : FreeBSD

- Log shows zero information yet epmd is eating 100% CPU constantly

- /usr/local/lib/erlang19/erts-8.3.5.4/bin/epmd

- It was a Thursday

- This is based on the public FreeBSD Amazon AMI

- Also, and very importantly, after about 10 minutes the AMI was able to come back up, so this time I didn’t go the route of completely reconstructing the system again

=INFO REPORT==== 27-Mar-2018::01:00:38 ===
Starting RabbitMQ 3.6.14 on Erlang 19.3.6.5
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

- Config is extremely vanilla. Just using

[
{rabbit,
{loopback_users, []},
{heartbeat, 60}
]},
{kernel,
]},
{rabbitmq_management,
]},
{rabbitmq_shovel,
[{shovels,
]}
]},
{rabbitmq_web_stomp, [{use_http_auth, true}]},
{rabbitmq_stomp,
{default_user, [{login, "guest"}, {passcode, "guest"}]},
{tcp_listeners, [{"127.0.0.1", 61613},
{"::1", 61613}]}
]},
{rabbitmq_mqtt,
]},
{rabbitmq_amqp1_0,
]},
{rabbitmq_auth_backend_ldap,
]}
].

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

How can we set up FreeBSD jails the same way your environment has? Can you reproduce the problem

and run dtrace on epmd?

I'm not a FreeBSD expert so I'm asking for as many details as possible. Our team is fortunate to have a FreeBSD committer

but even that person would need some specific steps to reproduce.

epmd consuming 100% of the CPU sound suspicious. If we can find out what it is doing we could at least report it to the Erlang/OTP team:

epmd is not a part of RabbitMQ.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
> To post to this group, send email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
> To post to this group, send email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Wedge Martin]

It’s the 11.1 version ( think it’s the only one ) in the public AMIs when you go to spin up an instance. Didn’t do much beyond that though I did enable web stomp.   If I had half a brain I would’ve done a truss on the epmd process to see what the hell it was churning on.  When it happens again, I”ll get a good solid trace instead of just lashing out like a teenager whose had his iPad taken away.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

So in other words, the steps are

 * Start a FreeBSD EC2 instance using a specific AMI

 * Ensure that RabbitMQ is provisioned… is it preinstalled in the AMI or should we install it from ports?

 * Enable rabbitmq-web-stomp

 * Reboot?

 * Wait? Or should we expect the symptoms to occur immediately?

I'm not sure I understand how FreeBSD jails are involved (they were specifically mentioned in the original post).

> To post to this group, send email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>

> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.