cannot_read_file error when starting rabbitmq-server on CentOS 6.5

[Fraser]

Hi all,

I am trying to get a RabbitMQ server running on CentOS 6.5. I have installed Erlang and the latest version of rabbitmq-server (3.3.5) using the instructions for RPM-based Linux, with no errors on install.

The user RabbitMQ is created, and I add him to the wheel usergroup to enable sudo use.

When I attempt to run

sudo service rabbitmq-server start

as the user rabbitmq, it fails.

[rabbitmq@localhost /]$ sudo service rabbitmq-server start

Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}

rabbitmq-server.

When I check the log file, I get an error for /var/lib/rabbitmq/rabbit_localhost/cluster_nodes.config or nodes_running_at_shutdown, which cannot be read:

[rabbitmq@localhost /]$ cat /var/log/rabbitmq/startup_log

WARNING: ignoring /etc/rabbitmq/rabbitmq.conf -- location has moved to /etc/rabbitmq/rabbitmq-env.conf

BOOT FAILED

===========

Error description:

   {error,{cannot_read_file,"/var/lib/rabbitmq/mnesia/rabbit_localhost/nodes_running_at_shutdown",

                            eacces}}

Log files (may contain more information):

   /var/log/rabbitmq/rabbit_localhost.log

   /var/log/rabbitmq/rabbit_localhost-sasl.log

Stack trace:

   [{rabbit_node_monitor,try_read_file,1},

    {rabbit_node_monitor,prepare_cluster_status_files,0},

    {rabbit,'-boot/0-fun-1-',0},

    {rabbit,start_it,1},

    {init,start_it,1},

    {init,start_em,1}]

{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{cannot_read_file,"/var/lib/rabbitmq/mnesia/rabbit_localhost/nodes_running_at_shutdown",eacces}}}}

So I go and change the permissions:

[rabbitmq@localhost /]$ sudo chmod -R 755 /var/lib/rabbitmq/mnesia/rabbit_localhost

[rabbitmq@localhost /]$ ls -la /var/lib/rabbitmq/mnesia/rabbit_localhost/

total 16

drwxr-xr-x. 2 rabbitmq rabbitmq 4096 Sep 16 17:02 .

drwxr-xr-x. 4 rabbitmq rabbitmq 4096 Sep 16 16:29 ..

-rwxr-xr-x. 1 rabbitmq rabbitmq   61 Sep 16 17:02 cluster_nodes.config

-rwxr-xr-x. 1 rabbitmq rabbitmq   30 Sep 16 17:02 nodes_running_at_shutdown

and start the service again:

[rabbitmq@localhost /]$ sudo service rabbitmq-server start

Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}

rabbitmq-server.

[rabbitmq@localhost /]$ cat /var/log/rabbitmq/startup_log

WARNING: ignoring /etc/rabbitmq/rabbitmq.conf -- location has moved to /etc/rabbitmq/rabbitmq-env.conf

BOOT FAILED

===========

Error description:

   {error,{cannot_read_file,"/var/lib/rabbitmq/mnesia/rabbit_localhost/cluster_nodes.config",

                            eacces}}

Log files (may contain more information):

   /var/log/rabbitmq/rabbit_localhost.log

   /var/log/rabbitmq/rabbit_localhost-sasl.log

Stack trace:

   [{rabbit_node_monitor,try_read_file,1},

    {rabbit_node_monitor,read_cluster_status,0},

    {rabbit_mnesia,cluster_status,1},

    {rabbit_upgrade,maybe_upgrade_mnesia,0},

    {rabbit,'-boot/0-fun-1-',0},

    {rabbit,start_it,1},

    {init,start_it,1},

    {init,start_em,1}]

{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{cannot_read_file,"/var/lib/rabbitmq/mnesia/rabbit_localhost/cluster_nodes.config",eacces}}}}

I get the same error. I check the file permissions and they have reset!:

[rabbitmq@localhost /]$ ls -la /var/lib/rabbitmq/mnesia/rabbit_localhost/

total 16

drwxr-xr-x. 2 rabbitmq rabbitmq 4096 Sep 16 17:26 .

drwxr-xr-x. 4 rabbitmq rabbitmq 4096 Sep 16 16:29 ..

----------. 1 rabbitmq rabbitmq   61 Sep 16 17:26 cluster_nodes.config

----------. 1 rabbitmq rabbitmq   30 Sep 16 17:26 nodes_running_at_shutdown

I've tried looking around in the rabbitmq-server and rabbitmq-env scripts but I can find nothing to explain why the file permissions are reverting to "none" every time I start the service.

Any help would be very much appreciated!

Thanks and kind regards,

Fraser

[Simon MacMullen]

On 16/09/14 17:31, Fraser wrote:
> Hi all,
>
> I am trying to get a RabbitMQ server running on CentOS 6.5. I have
> installed Erlang and the latest version of rabbitmq-server (3.3.5) using
> the instructions for RPM-based Linux

> <http://www.rabbitmq.com/install-rpm.html>, with no errors on install.

>
> The user RabbitMQ is created, and I add him to the wheel usergroup to
> enable sudo use.

That should not be necessary - the rabbitmq user should never need to
become root.

<snip>

> I get the same error. I check the file permissions and they have reset!:
>
> |
> [rabbitmq@localhost /]$ ls -la /var/lib/rabbitmq/mnesia/rabbit_localhost/
> total 16
> drwxr-xr-x. 2 rabbitmq rabbitmq 4096 Sep 16 17:26 .
> drwxr-xr-x. 4 rabbitmq rabbitmq 4096 Sep 16 16:29 ..
> ----------. 1 rabbitmq rabbitmq 61 Sep 16 17:26 cluster_nodes.config
> ----------. 1 rabbitmq rabbitmq 30 Sep 16 17:26 nodes_running_at_shutdown

That's... very odd.

> I've tried looking around in the rabbitmq-server and rabbitmq-env
> scripts but I can find nothing to explain why the file permissions are
> reverting to "none" every time I start the service.

Nothing in any RabbitMQ script changes ownership or permissions of anything.

Could something outside RabbitMQ be going and changing permissions?

Cheers, Simon

[Fraser]

Thanks for your reply Simon, it did help me to narrow down my search. I managed to find a solution, so I will post it here for anyone else who experiences the same problem!

Previously I was only editing permissions for the /var/lib/rabbitmq/mnesia directory. Instead, change the permissions of the /var/lib/rabbitmq/ directory:

[rabbitmq@localhost /]$ sudo chmod -R 755 /var/lib/rabbitmq/

And edit the permissions of the /var/lib/rabbitmq/.erlang.cookie file to be accessible by the owner only (this resolves another error on rabbitmq-server start):

[rabbitmq@localhost /]$ sudo chmod 600 /var/lib/rabbitmq/.erlang.cookie

Another file in /etc/rabbitmq also requires a permission change to 755, check the startup_log when you attempt to run rabbitmq-server start to find the file name. Afterwards, rabbitmq-server will start successfully!