We are having a rabbitmq cluster (2 nodes) running since quite a while and it works really fine. Just recently our company forced to restart our servers due to maintenance reasons.
Since then I am unable to start rabbitmq-server via "sudo service rabbitmq-server start" command. I am still able to start it via rabbitmq user and command "rabbitmq-server -detached" but
I don't really understand what is different and why does it work for rabbitmq user but not for the root user.
I was trying to dive into the startup script and realised that rabbitmqctl status is giving different return codes depending if I run it as root or as rabbitmq.
Any help which point me in a direction is greatly appreciated.
[sys_eventhub1@nrchbs-slt0038 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
[sys_eventhub1@nrchbs-slt0038 ~]$ yum list installed | grep rabbitmq
rabbitmq-server.noarch 3.6.0-1 @rabbitmq_rabbitmq-server
[sys_eventhub1@nrchbs-slt0038 ~]$ yum list installed | grep erlang.x86
erlang.x86_64 18.3-1.el6 @erlang_solutions
[sys_eventhub1@nrchbs-slt0038 ~]$ sudo service rabbitmq-server stop
Stopping rabbitmq-server: rmdir: failed to remove `/var/run/rabbitmq': No such file or directory
rabbitmq-server.
[sys_eventhub1@nrchbs-slt0038 ~]$ ps -ef | grep rabbit
93687439 8500 3485 0 23:37 pts/0 00:00:00 grep rabbit
rabbitmq 18012 1 0 Nov07 ? 00:00:37 /usr/lib64/erlang/erts-7.3/bin/epmd -daemon
[sys_eventhub1@nrchbs-slt0038 ~]$ sudo service rabbitmq-server start
Starting rabbitmq-server: RabbitMQ is currently running
rabbitmq-server.
[sys_eventhub1@nrchbs-slt0038 ~]$ sudo service rabbitmq-server status
Status of node 'rabbit@nrchbs-slt0038' ...
Error: unable to connect to node 'rabbit@nrchbs-slt0038': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@nrchbs-slt0038']
rabbit@nrchbs-slt0038:
* connected to epmd (port 4369) on nrchbs-slt0038
* epmd reports: node 'rabbit' not running at all
no other nodes on nrchbs-slt0038
* suggestion: start the node
current node details:
- node name: 'rabbitmq-cli-6538@nrchbs-slt0038'
- home dir: /var/lib/rabbitmq
- cookie hash: 3VSUcIMHTgxY2zqMy228jw==
[sys_eventhub1@nrchbs-slt0038 ~]$ sudo su - rabbitmq
-bash-4.1$ rabbitmq-server -detached
Warning: PID file not written; -detached was passed.
-bash-4.1$ ps -ef | grep rabbit
root 8545 3485 0 23:39 pts/0 00:00:00 sudo su - rabbitmq
root 8546 8545 0 23:39 pts/0 00:00:00 su - rabbitmq
rabbitmq 8547 8546 1 23:39 pts/0 00:00:00 -bash
rabbitmq 8841 1 99 23:39 ? 00:00:08 /usr/lib64/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.0/ebin -noshell -noinput -s rabbit boot -sname rabbit@nrchbs-slt0038 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@nrchbs-slt0038.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@nrchbs-slt0038-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.0/plugins" -rabbit plugins_expand_dir "/apps/rabbitmq/mnesia/rabbit@nrchbs-slt0038-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/apps/rabbitmq/mnesia/rabbit@nrchbs-slt0038" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672 -noshell -noinput
rabbitmq 8919 8841 0 23:39 ? 00:00:00 inet_gethost 4
rabbitmq 8920 8919 0 23:39 ? 00:00:00 inet_gethost 4
rabbitmq 8936 8547 1 23:39 pts/0 00:00:00 ps -ef
rabbitmq 8937 8547 0 23:39 pts/0 00:00:00 grep rabbit
rabbitmq 18012 1 0 Nov07 ? 00:00:37 /usr/lib64/erlang/erts-7.3/bin/epmd -daemon
[sys_eventhub1@nrchbs-slt0038 ~]$ sudo service rabbitmq-server status
Status of node 'rabbit@nrchbs-slt0038' ...
[{pid,7142},
{running_applications,
[{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.6.0"},
{rabbitmq_management,"RabbitMQ Management Console","3.6.0"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.0"},
{rabbit,"RabbitMQ","3.6.0"},
{os_mon,"CPO CXC 138 46","2.4"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.0"},
{webmachine,"webmachine","git"},
{mochiweb,"MochiMedia Web Server","2.13.0"},
{ssl,"Erlang/OTP SSL application","7.3"},
{public_key,"Public key infrastructure","1.1.1"},
{crypto,"CRYPTO","3.6.3"},
{amqp_client,"RabbitMQ AMQP Client","3.6.0"},
{rabbit_common,[],"3.6.0"},
{asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},
{syntax_tools,"Syntax tools","1.7"},
{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
{xmerl,"XML parser","1.3.10"},
{compiler,"ERTS CXC 138 10","6.0.3"},
{inets,"INETS CXC 138 49","6.2"},
{mnesia,"MNESIA CXC 138 12","4.13.3"},
{sasl,"SASL CXC 138 11","2.7"},
{stdlib,"ERTS CXC 138 10","2.8"},
{kernel,"ERTS CXC 138 10","4.2"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 18 [erts-7.3] [source-d2a6d81] [64-bit] [smp:6:6] [async-threads:64] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,61192760},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,2808},
{queue_procs,536624},
{queue_slave_procs,638768},
{plugins,567160},
{other_proc,19804088},
{mnesia,492720},
{mgmt_db,58584},
{msg_index,84672},
{other_ets,1479240},
{binary,244408},
{code,27352402},
{atom,992409},
{other_system,8938877}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
{vm_memory_high_watermark,0.6},
{vm_memory_limit,20201019801},
{file_descriptors,
[{total_limit,10140},
{total_used,4},
{sockets_limit,9124},
{sockets_used,0}]},
{processes,[{limit,1048576},{used,362}]},
{run_queue,0},
{uptime,52},
{kernel,{net_ticktime,60}}]