RabbitMQ waits forever for PID file

[César]

Hi all,

I tried following up an old thread about this ( http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-waits-forever-for-PID-file-during-startup-td36086.html ), but for some reason the email failed to be delivered, so I'm creating a new topic. Apologies for the repetition!

Basically, we came across this issue again. Our software has changed a bit though. We have upgraded RabbitMQ to version 3.5.4 and Erlang to version 17.5 (we thought upgrading Erlang might help us with memory allocation issues).

The process tree looks like this:

root     10785  0.0  0.0  11564  1672 ?        Ss   Nov30   0:00      \_ /bin/sh /sbin/service rabbitmq-server start

root     10790  0.0  0.0 108432  1808 ?        S    Nov30   0:00          \_ /bin/sh /etc/init.d/rabbitmq-server start

root     10946  0.0  0.0 108164  1456 ?        S    Nov30   0:00              \_ /bin/sh /usr/sbin/rabbitmqctl wait /var/run/rabbitmq/pid

root     10970  0.0  0.0 141356  1792 ?        S    Nov30   0:00                  \_ su rabbitmq -s /bin/sh -c /usr/lib64/rabbitmq/bin/rabbitmqctl  'wait' '/var/run/rabbitmq/pid'

rabbitmq 10973  0.1  0.0 2900308 128488 ?      Ssl  Nov30   1:13                      \_ /usr/lib64/erlang/erts-6.4/bin/beam.smp -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib64/rabbitmq/lib/rabbitmq_server-3.5.4/sbin/../ebin -noshell -noinput -hidden -boot start_clean -sasl errlog_type error -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@ms1dot90" -s rabbit_control_main -nodename rabbit@ms1dot90 -extra wait /var/run/rabbitmq/pid

And the error message in startup_err

[root@ms1dot90 ~]# cat /var/log/rabbitmq/startup_err

/usr/lib64/rabbitmq/bin/rabbitmq-server: line 42: /var/run/rabbitmq/pid: Cannot allocate memory

The referenced line is still echo $$ > ${RABBITMQ_PID_FILE}

I was wondering if there would be any way to make the execution fail straight away?

Thanks!

Cesar.

[Jean-Sébastien Pédron]

On 02/12/2015 16:23, César wrote:
> Hi all,

Hi!

> And the error message in startup_err
>
> [root@ms1dot90 ~]# cat /var/log/rabbitmq/startup_err
> /usr/lib64/rabbitmq/bin/rabbitmq-server: line 42: /var/run/rabbitmq/pid:
> Cannot allocate memory
>

> I was wondering if there would be any way to make the execution fail
> straight away?

"rabbitmqctl wait /path/to/pid" is designed to:
1. wait for the creation of the pidfile
2. wait for RabbitMQ to start in the node designated by this PID

We must do this because when "rabbitmqctl wait" is started, the pidfile
may not have been created yet. The problem with this design is we can't
distinguish this from a node which will not start at all (ie. no pidfile
is created).

"rabbitmqctl wait" could watch for the node to appear but the problem is
the same. Perhaps we could have a wrapper who records the fact that
"node $node should run in the near future" and clears this record when
the startup script returns, and rabbitmqct would watch this record. I
don't know if that would work. I will file an issue.

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[César]

Hi Jean-Sébastien,

that's great, thanks! I was wondering if there is any ticket reference we could use to follow up and know to which version of RabbitMQ we should upgrade once the issue is fixed?

Cheers,

Cesar.

[Jean-Sébastien Pédron]

On 03/12/2015 08:25, César wrote:
> Hi Jean-Sébastien,

Hi!

> that's great, thanks! I was wondering if there is any ticket reference
> we could use to follow up and know to which version of RabbitMQ we
> should upgrade once the issue is fixed?

Here is the ticket:
https://github.com/rabbitmq/rabbitmq-server/issues/463

[César]

We'll keep an eye on it :), thanks so much!