rabbitmq-server start failed

4,381 views

Skip to first unread message

Sam Lee

unread,

Feb 10, 2017, 4:07:09 AM2/10/17

to rabbitmq-users

when I run command "service rabbitmq-server start". the rabbitmq service not run correctly

[root@node-102 system]# uname -a

Linux node-102.domain.tld 3.10.0-327.20.1.es2.el7.x86_64 #1 SMP Sat May 21 16:37:14 CST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@node-102 system]#

[root@node-102 system]# cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

[root@node-102 system]# yum info rabbitmq-server

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

Installed Packages

Name : rabbitmq-server

Arch : noarch

Version : 3.6.5

Release : 1.el7

Size : 5.8 M

Repo : installed

From repo : nailgun

Summary : The RabbitMQ server

URL : http://www.rabbitmq.com/

License : MPLv1.1

Description : RabbitMQ is an implementation of AMQP, the emerging standard for high

: performance enterprise messaging. The RabbitMQ server is a robust and

: scalable implementation of an AMQP broker.

[root@node-102 ~]# service rabbitmq-server stop

Redirecting to /bin/systemctl stop rabbitmq-server.service

log in /var/log/rabbitmq/rab...@node-102.log

=INFO REPORT==== 10-Feb-2017::16:52:05 ===

Stopping RabbitMQ

=INFO REPORT==== 10-Feb-2017::16:52:05 ===

stopped TCP Listener on [::]:5673

=ERROR REPORT==== 10-Feb-2017::16:52:05 ===

Error on AMQP connection <0.728.0> (127.0.0.1:43394 -> 127.0.0.1:5673, vhost: '/', user: 'nova', state: running), channel 0:

operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"

=INFO REPORT==== 10-Feb-2017::16:52:05 ===

Stopped RabbitMQ application

=INFO REPORT==== 10-Feb-2017::16:52:05 ===

Halting Erlang VM

[root@node-102 ~]# service rabbitmq-server start

Redirecting to /bin/systemctl start rabbitmq-server.service

Job for rabbitmq-server.service failed because a timeout was exceeded. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

[root@node-102 ~]# service rabbitmq-server status -l

Redirecting to /bin/systemctl status -l rabbitmq-server.service

● rabbitmq-server.service - RabbitMQ broker

Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)

Drop-In: /etc/systemd/system/rabbitmq-server.service.d

└─limits.conf

Active: failed (Result: timeout) since Fri 2017-02-10 16:53:28 CST; 3min 9s ago

Process: 63835 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)

Process: 64339 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=killed, signal=TERM)

Main PID: 64339 (code=killed, signal=TERM)

Feb 10 16:52:28 node-102.domain.tld systemd[1]: Starting RabbitMQ broker...

Feb 10 16:52:29 node-102.domain.tld rabbitmq-server[64339]: =INFO REPORT==== 10-Feb-2017::16:52:29 ===

Feb 10 16:52:29 node-102.domain.tld rabbitmq-server[64339]: Ignoring external configuration due to error: enoent

Feb 10 16:53:28 node-102.domain.tld systemd[1]: rabbitmq-server.service start operation timed out. Terminating.

Feb 10 16:53:28 node-102.domain.tld systemd[1]: Failed to start RabbitMQ broker.

Feb 10 16:53:28 node-102.domain.tld systemd[1]: Unit rabbitmq-server.service entered failed state.

Feb 10 16:53:28 node-102.domain.tld systemd[1]: rabbitmq-server.service failed.

1. the command "service rabbitmq-server start "is unblocked when the timeout is reached .

2. The command "service rabbitmq-server start" gets unblocked if I issue "rabbitmqctl start_app" in another window.

I checked ps fax, there is nothing useful output.

[root@node-102 ~]# ps fax

......
75966 ? Ssl 0:00 /usr/lib64/erlang/erts-7.3.1.2/bin/beam.smp -W w -A 768 -K true -A30 -P 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/l
76322 ? Ss 0:00 \_ inet_gethost 4
76323 ? S 0:00 \_ inet_gethost 4

Please suggest how to gather more information on why the command fails to start the rabbitmq app.

Thanks in advance.

Michael Klishin

unread,

Feb 10, 2017, 4:09:01 AM2/10/17

to rabbitm...@googlegroups.com, Sam Lee

ENOENT means a path (file, directory) or something else does not exist.
It could be due to a CA or server certificate and key path typo, for example.

On 10 February 2017 at 12:07:13, Sam Lee (lixq...@gmail.com) wrote:
> when I run command "service rabbitmq-server start". the rabbitmq service
> not run correctly
>
>

> [root@node-102 system]# *uname -a*

> Linux node-102.domain.tld 3.10.0-327.20.1.es2.el7.x86_64 #1 SMP Sat May 21
> 16:37:14 CST 2016 x86_64 x86_64 x86_64 GNU/Linux
> [root@node-102 system]#

> [root@node-102 system]# *cat /etc/redhat-release*

> CentOS Linux release 7.2.1511 (Core)
>
>

> [root@node-102 system]# *yum info rabbitmq-server*

> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> Installed Packages
> Name : rabbitmq-server
> Arch : noarch
> Version : 3.6.5
> Release : 1.el7
> Size : 5.8 M
> Repo : installed
> From repo : nailgun
> Summary : The RabbitMQ server
> URL : http://www.rabbitmq.com/
> License : MPLv1.1
> Description : RabbitMQ is an implementation of AMQP, the emerging standard
> for high
> : performance enterprise messaging. The RabbitMQ server is a
> robust and
> : scalable implementation of an AMQP broker.
>
>

> [root@node-102 ~]# *service rabbitmq-server stop*

> Redirecting to /bin/systemctl stop rabbitmq-server.service
>
>

> *log in /var/log/rabbitmq/rab...@node-102.log*

>
> =INFO REPORT==== 10-Feb-2017::16:52:05 ===
> Stopping RabbitMQ
>
> =INFO REPORT==== 10-Feb-2017::16:52:05 ===
> stopped TCP Listener on [::]:5673
>
> =ERROR REPORT==== 10-Feb-2017::16:52:05 ===
> Error on AMQP connection <0.728.0> (127.0.0.1:43394 -> 127.0.0.1:5673,
> vhost: '/', user: 'nova', state: running), channel 0:
> operation none caused a connection exception connection_forced: "broker
> forced connection closure with reason 'shutdown'"
>
> =INFO REPORT==== 10-Feb-2017::16:52:05 ===
> Stopped RabbitMQ application
>
> =INFO REPORT==== 10-Feb-2017::16:52:05 ===
> Halting Erlang VM
>

> [root@node-102 ~]# *service rabbitmq-server start*

> Redirecting to /bin/systemctl start rabbitmq-server.service
> Job for rabbitmq-server.service failed because a timeout was exceeded. See
> "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
>
>

> [root@node-102 ~]# *service rabbitmq-server status -l*

> Redirecting to /bin/systemctl status -l rabbitmq-server.service
> ● rabbitmq-server.service - RabbitMQ broker
> Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service;
> enabled; vendor preset: disabled)
> Drop-In: /etc/systemd/system/rabbitmq-server.service.d
> └─limits.conf
> Active: failed (Result: timeout) since Fri 2017-02-10 16:53:28 CST; 3min
> 9s ago
> Process: 63835 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop
> (code=exited, status=0/SUCCESS)
> Process: 64339 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server
> (code=killed, signal=TERM)
> Main PID: 64339 (code=killed, signal=TERM)
>
> Feb 10 16:52:28 node-102.domain.tld systemd[1]: Starting RabbitMQ broker...
> Feb 10 16:52:29 node-102.domain.tld rabbitmq-server[64339]: =INFO
> REPORT==== 10-Feb-2017::16:52:29 ===

> Feb 10 16:52:29 node-102.domain.tld rabbitmq-server[64339]: *Ignoring
> external configuration due to error: enoent*

> Feb 10 16:53:28 node-102.domain.tld systemd[1]: rabbitmq-server.service
> start operation timed out. Terminating.
> Feb 10 16:53:28 node-102.domain.tld systemd[1]: Failed to start RabbitMQ
> broker.
> Feb 10 16:53:28 node-102.domain.tld systemd[1]: Unit
> rabbitmq-server.service entered failed state.
> Feb 10 16:53:28 node-102.domain.tld systemd[1]: rabbitmq-server.service
> failed.
>
>
>

> 1. the command "*service rabbitmq-server start *"is unblocked when the
> timeout is reached .
> 2. The command "*service rabbitmq-server start*" gets unblocked if I issue

> "rabbitmqctl start_app" in another window.
>
>
>

> I checked *ps fax, *there is nothing useful output.
>
> [root@node-102 ~]#
> *ps fax*
>
>
>
> *......75966 ? Ssl 0:00

> /usr/lib64/erlang/erts-7.3.1.2/bin/beam.smp -W w -A 768 -K true -A30 -P
> 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home
> /var/lib/rabbitmq -- -pa /usr/l 76322 ? Ss 0:00 \_
> inet_gethost 4 76323 ? S 0:00 \_ inet_gethost 4Please
> suggest how to gather more information on why the command fails to start

> the rabbitmq app. Thanks in advance.*
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Sam Lee

unread,

Feb 10, 2017, 4:27:47 AM2/10/17

to rabbitmq-users, lixq...@gmail.com

Any useful infomation to show which file or directory does not exist?

在 2017年2月10日星期五 UTC+8下午5:09:01，Michael Klishin写道：

> *log in /var/log/rabbitmq/rabbit@node-102.log*

Michael Klishin

unread,

Feb 10, 2017, 4:31:42 AM2/10/17

to rabbitm...@googlegroups.com

According to the message you are using rabbitmq-clusterer and configured

it to use an external config file, which cannot be loaded because of ENOENT:

https://github.com/rabbitmq/rabbitmq-clusterer/search?utf8=%E2%9C%93&q=Ignoring+external+configuration

log in /var/log/rabbitmq/rabbit@node-102.log

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Sam Lee

unread,

Feb 10, 2017, 5:05:55 AM2/10/17

to rabbitmq-users

[root@node-102 ~]# ps -ef | grep beam

rabbitmq 85971 1 8 17:13 ? 00:02:07 /usr/lib64/erlang/erts-7.3.1.2/bin/beam.smp -W w -A 768 -K true -A30 -P 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/ebin -noshell -noinput -s rabbit_clusterer boot -sname rabbit@node-102 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/plugins/rabbitmq_clusterer-3.6.x-667f92b0.ez/rabbitmq_clusterer-3.6.x-667f92b0/ebin -rabbit tcp_listeners [{"auto",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@node-102.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@node-102-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-102-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-102"

I don't configure any external config file. I check the /etc/rabbitmq/rabbit.conf

[root@node-102 rabbitmq]# cat /etc/rabbitmq/rabbitmq.config_bak

% This file managed by Puppet 3.8.7

% Template Path: rabbitmq/templates/rabbitmq.config

[

{kernel, [

{inet_dist_listen_min, 41055},

{inet_dist_listen_max, 41055},

{inet_default_connect_options, [{nodelay,true}]}

]},

{rabbit, [

{log_levels, [{connection, info}]},

{default_vhost, <<"/">>},

{default_user, <<"">>},

{default_pass, <<"">>},

{default_permissions, [<<".*">>, <<".*">>, <<".*">>]},

{cluster_partition_handling, autoheal},

{tcp_listen_options, [

binary,

{packet, raw},

{reuseaddr, true},

{backlog, 128},

{nodelay, true},

{exit_on_close, false},

{keepalive, true}

]}

,{rabbitmq_clusterer,

[{config, "/etc/rabbitmq/cluster.config"}]

}

% EOF

the cluster config file cluster.config is not in /etc/rabbitmq directory, in fact in /var/lib/rabbitmq/mnesia directory. I change this and restart rabbitmq-server, the rabbitmq-server still can not be start correctly。

I check the other nodes where run correctly in this cluster. the cluster config in rabbitmq.config, they are "/etc/rabbitmq/cluster.config" too.

在 2017年2月10日星期五 UTC+8下午5:31:42，Michael Klishin写道：

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,

Feb 10, 2017, 5:42:20 AM2/10/17

to rabbitm...@googlegroups.com

And what is /etc/rabbitmq/cluster.config in your config file if not an external clusterer configuration file?

On 10 Feb 2017, at 13:05, Sam Lee <lixq...@gmail.com> wrote:

/etc/rabbitmq/cluster.config

Sam Lee

unread,

Feb 12, 2017, 8:38:12 PM2/12/17

to rabbitmq-users

this is a three nodes cluster, node-100, node-102, node-108. The file /etc/rabbitmq/cluster.config in node-102 and node-108 doesn't exist. In node-100 the content of file /etc/rabbitmq
/cluster.config is as below:

[

{version, 1},

{nodes, [

{'rabbit@node-100', disc},{'rabbit@node-102', disc},{'rabbit@node-108', disc}

]},

{gospel, {node, 'rabbit@node-100'}}

node-100 and node-108 can start rabbitmq-server correctly. node-102 will hang

在 2017年2月10日星期五 UTC+8下午6:42:20，Michael Klishin写道：

Michael Klishin

unread,

Feb 12, 2017, 9:23:38 PM2/12/17

to rabbitm...@googlegroups.com

Either create the external clusterer config file or use inline configuration.

All nodes that use rabbitmq-clusterer must be configured the same way. If an external

config file is specified, it must exist and be valid.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Sam Lee

unread,

Feb 13, 2017, 2:29:14 AM2/13/17

to rabbitmq-users

create the same file in node-102 and node-108, the service still can't start correctly. use strace to trace the system call

[root@node-102 rabbitmq]# strace /usr/sbin/rabbitmq-server

execve("/usr/sbin/rabbitmq-server", ["/usr/sbin/rabbitmq-server"], [/* 22 vars */]) = 0

brk(0) = 0x2079000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3946be000

access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=35970, ...}) = 0

mmap(NULL, 35970, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb3946b5000

close(3) = 0

open("/lib64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\316\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=174520, ...}) = 0

mmap(NULL, 2268928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb394274000

mprotect(0x7fb394299000, 2097152, PROT_NONE) = 0

mmap(0x7fb394499000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7fb394499000

close(3) = 0

open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=19520, ...}) = 0

mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb394070000

mprotect(0x7fb394073000, 2093056, PROT_NONE) = 0

mmap(0x7fb394272000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb394272000

close(3) = 0

open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \34\2\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=2107816, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3946b4000

mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb393caf000

mprotect(0x7fb393e65000, 2097152, PROT_NONE) = 0

mmap(0x7fb394065000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7fb394065000

mmap(0x7fb39406b000, 16960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb39406b000

close(3) = 0

mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3946b2000

arch_prctl(ARCH_SET_FS, 0x7fb3946b2740) = 0

mprotect(0x7fb394065000, 16384, PROT_READ) = 0

mprotect(0x7fb394272000, 4096, PROT_READ) = 0

mprotect(0x7fb394499000, 16384, PROT_READ) = 0

mprotect(0x6dc000, 4096, PROT_READ) = 0

mprotect(0x7fb3946bf000, 4096, PROT_READ) = 0

munmap(0x7fb3946b5000, 35970) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

open("/dev/tty", O_RDWR|O_NONBLOCK) = 3

close(3) = 0

brk(0) = 0x2079000

brk(0x209a000) = 0x209a000

brk(0) = 0x209a000

open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=106065056, ...}) = 0

mmap(NULL, 106065056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb38d788000

close(3) = 0

brk(0) = 0x209a000

getuid() = 0

getgid() = 0

geteuid() = 0

getegid() = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3946bd000

read(3, "MemTotal: 131413992 kB\nMem"..., 1024) = 1024

close(3) = 0

munmap(0x7fb3946bd000, 4096) = 0

rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {SIG_DFL, [], 0}, 8) = 0

rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], 0}, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], 0}, 8) = 0

rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

uname({sys="Linux", node="node-102.domain.tld", ...}) = 0

stat("/etc/rabbitmq", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

getpid() = 3517248

open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=26254, ...}) = 0

mmap(NULL, 26254, PROT_READ, MAP_SHARED, 3, 0) = 0x7fb3946b7000

close(3) = 0

getppid() = 3517245

getpgrp() = 3517245

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

getrlimit(RLIMIT_NPROC, {rlim_cur=513242, rlim_max=513242}) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

open("/usr/sbin/rabbitmq-server", O_RDONLY) = 3

ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff4bb1dd90) = -1 ENOTTY (Inappropriate ioctl for device)

lseek(3, 0, SEEK_CUR) = 0

read(3, "#!/bin/sh\n## The contents of th"..., 80) = 80

lseek(3, 0, SEEK_SET) = 0

getrlimit(RLIMIT_NOFILE, {rlim_cur=100*1024, rlim_max=110*1024}) = 0

fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor)

dup2(3, 255) = 255

close(3) = 0

fcntl(255, F_SETFD, FD_CLOEXEC) = 0

fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)

fstat(255, {st_mode=S_IFREG|0755, st_size=1714, ...}) = 0

lseek(255, 0, SEEK_CUR) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

read(255, "#!/bin/sh\n## The contents of th"..., 1714) = 1714

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

lseek(255, -961, SEEK_CUR) = 753

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517249

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "Linux\n", 128) = 6

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517249, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517249

wait4(-1, 0x7fff4bb1d110, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 6

read(3, "", 128) = 0

close(3) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

read(255, "\nfor arg in \"$@\" ; do\n # Wrap"..., 1714) = 961

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

stat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

stat("/var/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

stat("/var/lib/rabbitmq", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0

chdir("/var/lib/rabbitmq") = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

lseek(255, -677, SEEK_CUR) = 1037

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517252

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "rabbitmq-server\n", 128) = 16

read(3, "", 128) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517252, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517252

wait4(-1, 0x7fff4bb1d250, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

close(3) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

read(255, "\nif [ `id -u` = `id -u rabbitmq`"..., 1714) = 677

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517253

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "0\n", 128) = 2

read(3, "", 128) = 0

close(3) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517253, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517253

wait4(-1, 0x7fff4bb1d250, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517254

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "983\n", 128) = 4

read(3, "", 128) = 0

close(3) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517254, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517254

wait4(-1, 0x7fff4bb1d250, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517255

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "0\n", 128) = 2

read(3, "", 128) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517255, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517255

wait4(-1, 0x7fff4bb1d110, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

close(3) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517256

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "983\n", 128) = 4

read(3, "", 128) = 0

close(3) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3517256

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517256, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, 0x7fff4bb1d010, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

pipe([3, 4]) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517258

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGCHLD, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, {0x441170, [], SA_RESTORER|SA_RESTART, 0x7fb393ce4670}, 8) = 0

close(4) = 0

read(3, "0\n", 128) = 2

read(3, "", 128) = 0

close(3) = 0

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3517258, si_status=0, si_utime=0, si_stime=0} ---

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 3517258

wait4(-1, 0x7fff4bb1d050, WNOHANG, NULL) = -1 ECHILD (No child processes)

rt_sigreturn() = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

stat(".", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0

stat("/usr/local/sbin/su", 0x7fff4bb1d720) = -1 ENOENT (No such file or directory)

stat("/usr/local/bin/su", 0x7fff4bb1d720) = -1 ENOENT (No such file or directory)

stat("/usr/sbin/su", 0x7fff4bb1d720) = -1 ENOENT (No such file or directory)

stat("/usr/bin/su", {st_mode=S_IFREG|S_ISUID|0755, st_size=32072, ...}) = 0

geteuid() = 0

getegid() = 0

getuid() = 0

getgid() = 0

access("/usr/bin/su", X_OK) = 0

stat("/usr/bin/su", {st_mode=S_IFREG|S_ISUID|0755, st_size=32072, ...}) = 0

geteuid() = 0

getegid() = 0

getuid() = 0

getgid() = 0

access("/usr/bin/su", R_OK) = 0

stat("/usr/bin/su", {st_mode=S_IFREG|S_ISUID|0755, st_size=32072, ...}) = 0

geteuid() = 0

getegid() = 0

getuid() = 0

getgid() = 0

access("/usr/bin/su", X_OK) = 0

stat("/usr/bin/su", {st_mode=S_IFREG|S_ISUID|0755, st_size=32072, ...}) = 0

geteuid() = 0

getegid() = 0

getuid() = 0

getgid() = 0

access("/usr/bin/su", R_OK) = 0

rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0

rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb3946b2a10) = 3517259

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

rt_sigaction(SIGINT, {0x43e5e0, [], SA_RESTORER, 0x7fb393ce4670}, {SIG_DFL, [], SA_RESTORER, 0x7fb393ce4670}, 8) = 0

wait4(-1, 0x7fff4bb1d820, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL, si_value={int=0, ptr=0x7ff600000000}} ---

wait4(-1, 0x7fff4bb1d820, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL, si_value={int=0, ptr=0x7ff600000000}} ---

wait4(-1, 0x7fff4bb1d820, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL, si_value={int=0, ptr=0x7ff600000000}} ---

wait4(-1, 0x7fff4bb1d820, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL, si_value={int=0, ptr=0x7ff600000000}} ---

wait4(-1,

在 2017年2月13日星期一 UTC+8上午10:23:38，Michael Klishin写道：

Either create the external clusterer config file or use inline configuration.

All nodes that use rabbitmq-clusterer must be configured the same way. If an external
config file is specified, it must exist and be valid.

On Mon, Feb 13, 2017 at 4:38 AM, Sam Lee <lixq...@gmail.com> wrote:

this is a three nodes cluster, node-100, node-102, node-108. The file /etc/rabbitmq/cluster.config in node-102 and node-108 doesn't exist. In node-100 the content of file /etc/rabbitmq
/cluster.config is as below:

[
{version, 1},
{nodes, [
{'rabbit@node-100', disc},{'rabbit@node-102', disc},{'rabbit@node-108', disc}
]},
{gospel, {node, 'rabbit@node-100'}}
].

node-100 and node-108 can start rabbitmq-server correctly. node-102 will hang

在 2017年2月10日星期五 UTC+8下午6:42:20，Michael Klishin写道：
And what is /etc/rabbitmq/cluster.config in your config file if not an external clusterer configuration file?

On 10 Feb 2017, at 13:05, Sam Lee <lixq...@gmail.com> wrote:

/etc/rabbitmq/cluster.config

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,

Feb 13, 2017, 5:47:28 AM2/13/17

to rabbitm...@googlegroups.com

rabbitmq-clusterer requires identical configuration deployed to all nodes.

If you use an external config file, that file must be present and valid.

You can inline rabbitmq-clusterer configuration into the regular rabbitmq.config file.

The error clearly comes from the rabbitmq-clusterer plugin and means that it

cannot read its external config file.

I don't have much to add to the above and not sure what you are implying by bolding out some lines in strace output.

Please take this from here on your own.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.