Rabbitmq Server looks like it started but error in sasl.log

Oi Lee

unread,

Oct 26, 2015, 2:16:27 PM10/26/15

to rabbitmq-users

I've been out of the office for about 2 weeks and when I come back my local CENTOS VM with Rabbitmq server is not working properly.

I start the server using: sudo /sbin/service rabbitmq-server start

It returns:

Starting rabbitmq-server (via systemctl): [ OK ]

I run: sudo rabbitmqctl status

Status of node rabbit@newDev ...

Error: unable to connect to node rabbit@newDev: nodedown

DIAGNOSTICS

===========

attempted to contact: [rabbit@newDev]

rabbit@newDev:

* connected to epmd (port 4369) on newDev

* epmd reports: node 'rabbit' not running at all

no other nodes on newDev

* suggestion: start the node

current node details:

- node name: 'rabbitmq-cli-15525@newDev'

- home dir: /var/lib/rabbitmq

- cookie hash: apkGvnTEZ9ucP+BQKDpFuQ==

I check the logs and found:

=CRASH REPORT==== 26-Oct-2015::12:09:53 ===

crasher:

initial call: tcp_listener:init/1

pid: <0.254.0>

registered_name: []

exception exit: {cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}

in function gen_server:init_it/6 (gen_server.erl, line 322)

ancestors: [<0.252.0>,rabbit_stomp_sup,<0.249.0>]

messages: []

links: [<0.252.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 233

stack_size: 27

reductions: 490

neighbours:

=SUPERVISOR REPORT==== 26-Oct-2015::12:09:53 ===

Supervisor: {<0.252.0>,tcp_listener_sup}

Context: start_error

Reason: {cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}

Offender: [{pid,undefined},

{name,tcp_listener},

{mfargs,

{tcp_listener,start_link,

[{0,0,0,0,0,0,0,0},

61613,

[inet6,binary,

{packet,raw},

{reuseaddr,true},

{backlog,128},

{nodelay,true}],

1,'tcp_acceptor_sup_:::61613',

{rabbit_networking,tcp_listener_started,[stomp]},

{rabbit_networking,tcp_listener_stopped,[stomp]},

"STOMP TCP Listener"]}},

{restart_type,transient},

{shutdown,4294967295},

{child_type,worker}]

=SUPERVISOR REPORT==== 26-Oct-2015::12:09:53 ===

Supervisor: {local,rabbit_stomp_sup}

Context: start_error

Reason: {shutdown,

{failed_to_start_child,tcp_listener,

{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}

Offender: [{pid,undefined},

{name,'rabbit_stomp_listener_sup_:::61613'},

{mfargs,

{tcp_listener_sup,start_link,

[{0,0,0,0,0,0,0,0},

61613,

[inet6,binary,

{packet,raw},

{reuseaddr,true},

{backlog,128},

{nodelay,true}],

{rabbit_networking,tcp_listener_started,[stomp]},

{rabbit_networking,tcp_listener_stopped,[stomp]},

{rabbit_stomp_sup,start_client,

[{stomp_configuration,"guest","guest",false,

false}]},

"STOMP TCP Listener"]}},

{restart_type,transient},

{shutdown,infinity},

{child_type,supervisor}]

=CRASH REPORT==== 26-Oct-2015::12:09:53 ===

crasher:

initial call: application_master:init/4

pid: <0.248.0>

registered_name: []

exception exit: {{shutdown,

{failed_to_start_child,

'rabbit_stomp_listener_sup_:::61613',

{shutdown,

{failed_to_start_child,tcp_listener,

{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}}},

{rabbit_stomp,start,[normal,[]]}}

in function application_master:init/4 (application_master.erl, line 133)

ancestors: [<0.247.0>]

messages: [{'EXIT',<0.249.0>,normal}]

links: [<0.247.0>,<0.7.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 376

stack_size: 27

reductions: 105

neighbours:

Not sure what happened or what I am doing incorrectly.

Thanks in advance for any help on this.

Michael Klishin

unread,

Oct 26, 2015, 2:26:28 PM10/26/15

to rabbitm...@googlegroups.com

RabbitMQ cannot bind to port 61613: see if SELinux or a firewall allow it to do so).

MK

Oi Lee

unread,

Oct 26, 2015, 2:55:33 PM10/26/15

to rabbitmq-users

Hi MK,

I test the ports using ServerSocket object in Java to see if it is closed. It is able to bind to the ports with no errors.

Thank.

Oi

Michael Klishin

unread,

Oct 26, 2015, 3:04:16 PM10/26/15

to rabbitm...@googlegroups.com

Did your test program also run with the same effective user as RabbitMQ?

Sorry but the error message is unambiguous: something prevents STOM plugin from binding to its non-TLS port.

See system logs (syslog and so on) for
possible clues.

Oi Lee

unread,

Oct 30, 2015, 2:19:50 PM10/30/15

to rabbitmq-users

Hi MK, no it does not run with the same user as RabbitMQ.

I removed everything and tried to start of scratch. I actually got startup_err and startup_log files today.

startup_err:

/usr/lib/rabbitmq/bin/rabbitmq-server: line 164: 14369 User defined signal 2 start_rabbitmq_server "$@"

startup_log:

## ## Licensed under the MPL. See http://www.rabbitmq.com/

## ##

########## Logs: /var/log/rabbitmq/rab...@newDev.log

###### ## /var/log/rabbitmq/rab...@newDev-sasl.log

##########

Starting broker...

BOOT FAILED

===========

Error description:

{could_not_start,rabbitmq_stomp,

{{shutdown,

{failed_to_start_child,'rabbit_stomp_listener_sup_:::61613',

{shutdown,

{failed_to_start_child,tcp_listener,

{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}}},

{rabbit_stomp,start,[normal,[]]}}}

Log files (may contain more information):

/var/log/rabbitmq/rab...@newDev.log

/var/log/rabbitmq/rab...@newDev-sasl.log

{"init terminating in do_boot",{could_not_start,rabbitmq_stomp,{{shutdown,{failed_to_start_child,'rabbit_stomp_listener_sup_:::61613',{shutdown,{failed_to_start_child,tcp_listener,{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}}},{rabbit_stomp,start,[normal,[]]}}}}

[ OK ]

Thanks!

Oi

Robert Raschke

unread,

Oct 30, 2015, 3:05:01 PM10/30/15

to rabbitm...@googlegroups.com

Looks like the user running Rabbit MQ does not have the appropriate permissions to listen on the Stomp port 61613.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Oi Lee

unread,

Oct 30, 2015, 3:20:35 PM10/30/15

to rabbitmq-users

I am really new to all this. How do you give RabbitMQ permissions? I never had to manually do it before until it stopped working.

On Friday, October 30, 2015 at 1:05:01 PM UTC-6, Robby Raschke wrote:

Looks like the user running Rabbit MQ does not have the appropriate permissions to listen on the Stomp port 61613.

On Oct 30, 2015 7:19 PM, "Oi Lee" <lee...@gmail.com> wrote:

Hi MK, no it does not run with the same user as RabbitMQ.

I removed everything and tried to start of scratch. I actually got startup_err and startup_log files today.

startup_err:
/usr/lib/rabbitmq/bin/rabbitmq-server: line 164: 14369 User defined signal 2 start_rabbitmq_server "$@"

startup_log:

RabbitMQ 3.5.6. Copyright (C) 2007-2015 Pivotal Software, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##

########## Logs: /var/log/rabbitmq/rabbit@newDev.log
###### ## /var/log/rabbitmq/rabbit@newDev-sasl.log

##########
Starting broker...

BOOT FAILED
===========

Error description:
{could_not_start,rabbitmq_stomp,
{{shutdown,
{failed_to_start_child,'rabbit_stomp_listener_sup_:::61613',
{shutdown,
{failed_to_start_child,tcp_listener,
{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}}},
{rabbit_stomp,start,[normal,[]]}}}

Log files (may contain more information):

/var/log/rabbitmq/rabbit@newDev.log
/var/log/rabbitmq/rabbit@newDev-sasl.log

{"init terminating in do_boot",{could_not_start,rabbitmq_stomp,{{shutdown,{failed_to_start_child,'rabbit_stomp_listener_sup_:::61613',{shutdown,{failed_to_start_child,tcp_listener,{cannot_listen,{0,0,0,0,0,0,0,0},61613,eacces}}}}},{rabbit_stomp,start,[normal,[]]}}}}
[ OK ]

Thanks!
Oi
On Monday, October 26, 2015 at 1:04:16 PM UTC-6, Michael Klishin wrote:
Did your test program also run with the same effective user as RabbitMQ?

Sorry but the error message is unambiguous: something prevents STOM plugin from binding to its non-TLS port.

See system logs (syslog and so on) for
possible clues.

> On 27 oct 2015, at 3:55, Oi Lee <lee...@gmail.com> wrote:
>
> I test the ports using ServerSocket object in Java to see if it is closed. It is able to bind to the ports with no errors.

Oi Lee

unread,

Oct 30, 2015, 4:39:02 PM10/30/15

to rabbitmq-users

So I found the SELInux Troubleshooter. At the time of the above error log I found that SELinux blocked erlang/erts-6.3/bin/beam.smp from name_bind access on the tcp_socket port 10181.

I'm unsure why beam.smp needed to access port 101081 or what beam.smp is. This was not one of the ports that was identified in the installation guide

Michael Klishin

unread,

Oct 31, 2015, 2:29:40 AM10/31/15

to rabbitm...@googlegroups.com, Oi Lee

On 30 October 2015 at 23:39:05, Oi Lee (lee...@gmail.com) wrote:
> I'm unsure why beam.smp needed to access port 101081 or what
> beam.smp is.

The Erlang VM.

> This was not one of the ports that was identified
> in the installation guide

The installation guide only covers *server* socket ports. Client socket ports
are ephemeral (randomly allocated from a known range). We cannot know their
values ahead of time.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Oi Lee

unread,

Nov 6, 2015, 2:17:18 PM11/6/15

to rabbitmq-users, lee...@gmail.com

That makes sense. Thanks! :)

Reply all

Reply to author

Forward