RabbitMQ fails to restart because of "eaddrinuse" error

[Zhen Q]

Hi,

We are running three rabbitmq nodes in one cluster. Sometimes rabbitmq fails to start after gracefully stop the service because of this error:

=CRASH REPORT==== xxx ====

  crasher:

    initial call: application_master:init/4

    pid: <0.1598.0>

    registered_name: []

    exception exit: {bad_return,

                        {{rabbit_mgmt_app,start,[normal,[]]},

                         {'EXIT',

                             {{could_not_start_listener,

                                  [{port,55672}],

                                  eaddrinuse},

                              {gen_server,call,

                                  [rabbit_web_dispatch_registry,

                                   {add,rabbit_mgmt,

                                       [{port,55672}],

                                       #Fun<rabbit_web_dispatch.1.31447083>,

                                       #Fun<rabbit_mgmt_app.0.15521781>,

                                       {[],"RabbitMQ Management"}},

                                   infinity]}}}}}

      in function  application_master:init/4

    ancestors: [<0.1597.0>]

    messages: [{'EXIT',<0.1599.0>,normal}]

    links: [<0.1597.0>,<0.7.0>]

    dictionary: []

    trap_exit: true

    status: running

    heap_size: 2584

    stack_size: 24

    reductions: 329

  neighbours:

Steps:

1. service rabbitmq-server stop

2. service rabbitmq-server start

RabbitMQ version: 3.5.1

Erlang version: R14B04

Return value of plugins:

$ rabbitmq-plugins list

 Configured: E = explicitly enabled; e = implicitly enabled

 | Status:   * = running on rabbit@mcp1

 |/

[e*] amqp_client                       3.5.1

[  ] cowboy                            0.5.0-rmq3.5.1-git4b93c2d

[  ] eldap                             3.5.1-gite309de4

[e*] mochiweb                          2.7.0-rmq3.5.1-git680dba8

[  ] rabbitmq_amqp1_0                  3.5.1

[  ] rabbitmq_auth_backend_ldap        3.5.1

[  ] rabbitmq_auth_mechanism_ssl       3.5.1

[  ] rabbitmq_consistent_hash_exchange 3.5.1

[  ] rabbitmq_federation               3.5.1

[  ] rabbitmq_federation_management    3.5.1

[E*] rabbitmq_management               3.5.1

[e*] rabbitmq_management_agent         3.5.1

[  ] rabbitmq_management_visualiser    3.5.1

[  ] rabbitmq_mqtt                     3.5.1

[  ] rabbitmq_shovel                   3.5.1

[  ] rabbitmq_shovel_management        3.5.1

[  ] rabbitmq_stomp                    3.5.1

[  ] rabbitmq_test                     3.5.1

[  ] rabbitmq_tracing                  3.5.1

[e*] rabbitmq_web_dispatch             3.5.1

[  ] rabbitmq_web_stomp                3.5.1

[  ] rabbitmq_web_stomp_examples       3.5.1

[  ] sockjs                            0.3.4-rmq3.5.1-git3132eb9

[e*] webmachine                        1.10.3-rmq3.5.1-gite9359c7

rabbitmq.config:

[

  {rabbit, [{cluster_nodes, {['rabbit@node1', 'rabbit@node2', 'rabbit@node3'], disc}},

            {cluster_partition_handling, 'autoheal'},

            {tcp_listen_options, [binary,

                                 {packet, raw},

                                 {reuseaddr, true},

                                 {backlog, 128},

                                 {nodelay, true},

                                 {exit_on_close, false},

                                 {keepalive, true}]}

           ]

  },

  {rabbitmq_mochiweb, [{listeners, [{mgmt, [{port, 55672}]}]}]},

  {rabbitmq_management, [{listener, [{port, 55672}]}]}

].

Do you think does the port setting on rabbitmq_mochiweb cause the issue? Should we remove it?

Thanks!

Zhen

[Michael Klishin]

 On 10 December 2015 at 20:51:14, Zhen Q (zqi...@gmail.com) wrote:
> {{could_not_start_listener,
>
>
> [{port,55672}],
>
>
> eaddrinuse},

This means that port 55672 is still in use.

We highly recommend upgrading to 3.5.6, 3.5.1 isn't going to get any updates..
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Zhen Qin]

Hi Michael,

Thanks for quick response! When this issue happened, we did check that management port 55672, but didn't find any app use it. That's why I am thinking if the configuration is wrong, because from the rabbit website:

"rabbitmq-mochiweb plugin provides hosting for other plugins that have HTTP interfaces. It allows these interfaces to co-exist on one or more HTTP listeners."

Is it possible other plugin using this port boot up earlier than rabbitmq-management?

Thanks!

Zhen

[Michael Klishin]

On 10 December 2015 at 22:41:51, Zhen Qin (zqi...@gmail.com) wrote:
> Thanks for quick response! When this issue happened, we did
> check that management port 55672, but didn't find any app use
> it. That's why I am thinking if the configuration is wrong, because
> from the rabbit website:
> "rabbitmq-mochiweb plugin provides hosting for other plugins
> that have HTTP interfaces. It allows these interfaces to co-exist
> on one or more HTTP listeners."
>
> Is it possible other plugin using this port boot up earlier than
> rabbitmq-management?

Wow, that page must be ancient.

Mochiweb is a Web server that backs the management plugin. So it tries to start
before its previous "instance" has finished stopping.

We are in the process of replacing Mochiweb with Cowboy. This won't be out until 3.7, however,
there's only so much time we'd be willing to put into fixing Mochiweb-specific issues. 

[David Shaw]

Interestingly, I have exactly the same error on 3.7.6

https://serverfault.com/questions/916650/rabbitmq-start-app-always-fails-service-restart-is-ok

The stop_app, reset, start_app will not work and complains about the port currently being used.  Oddly, a server restart or service restart and the services come up.  There must be something in the config but I'm damned if I can find it.  (I should add my MQ experience is at about the 3 days mark)

Thanks

Op donderdag 10 december 2015 23:13:23 UTC+1 schreef Michael Klishin:

[Michael Klishin]

Note that stop_app, start_app will not stop the OS process (the Erlang VM), which is what you

typically want when you run into EADDINUSE. There is nothing version-specific about it.

Use package-provided tools for managing services. Why you ended up having two instances

we can't know. Sometimes we see developers have multiple messaging brokers installed and they conflict

on a protocol port or two.

Use netstat or ss to find out [1]. And please start new threads for new questions in the future.

1. https://www.rabbitmq.com/troubleshooting-networking.html

The information contained in this e-mail communication is solely intended for the person/legal person to whom it has been sent, and as it may contain information of a personal or confidential nature, it may not be made public by virtue of law, regulations or agreement. If someone other than the intended recipient should receive or come into possession of this e-mail communication, he/she will not be entitled to read, disseminate, disclose or duplicate it. If you are not the intended recipient, you are requested to inform the sender of this e-mail message of this immediately, and to destroy the original e-mail communication. Neither Randstad Holding nv nor its subsidiaries accept any liability for incorrect and incomplete transmission or delayed receipt of this e-mail.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Forgot to mention that with generic UNIX packages (binary builds), `rabbitmqctl stop`

is the closest there is to a service management command. It will stop the runtime.

Starting a node back has to be done manually, with `./sbin/rabbitmq-server` or some wrapper

around it.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Luke Bakken]

Hi David,

I can't reproduce this on my workstation using RMQ 3.7.6 and a two-node cluster. You don't say exactly how you are executing stop_app, reset, and start_app and why - could you expand on that?

Thanks -

Luke

[David Shaw]

Thanks for the replies everyone.

I seem to have found the issue.  The setup I inherited had the management service bound to 35672.  Reading the docs, it looks like this is a reserved port:

35672-35682: used by CLI tools (Erlang distribution client ports) for communication with nodes and is allocated from a dynamic range (computed as server distribution port + 10000 through server distribution port + 10010). See networking guide for details.

I can only assume the startup order at boot / service restart allows the app to get the port before the reservation takes place - so the software works.  When you stop the app, reset and try to start it, the order must be slightly different which shows me the port going from closed to open (briefly) before the address in use error and closing again. i.e. two things are trying to bind to it.

I've set this to 15672

  {rabbitmq_management, [
    {listener, [
      {port, 15672}
    ]}
  ]}

and all appears well.  That was a pig to find, more so that it appeared to work in some situations.

Thanks for looking guys, I appreciate it.

Dave

Op vrijdag 15 juni 2018 17:59:27 UTC+2 schreef Luke Bakken:

[Michael Klishin]

Thank you for reporting back. I hope the Troubleshooting Networking guide was useful: it describes

some tools that can reveal what ports a process is bound to.

Some parts of the system initialize concurrently and do not assume that they can be sharing a port. That's why

the behavior isn't reproducible the same way every time: plugin activation does not synchronize with certain other parts.

The information contained in this e-mail communication is solely intended for the person/legal person to whom it has been sent, and as it may contain information of a personal or confidential nature, it may not be made public by virtue of law, regulations or agreement. If someone other than the intended recipient should receive or come into possession of this e-mail communication, he/she will not be entitled to read, disseminate, disclose or duplicate it. If you are not the intended recipient, you are requested to inform the sender of this e-mail message of this immediately, and to destroy the original e-mail communication. Neither Randstad Holding nv nor its subsidiaries accept any liability for incorrect and incomplete transmission or delayed receipt of this e-mail.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[David Shaw]

I appreciate others trying to help as well.

Hopefully I can play with MQ a little now and get to know it some more.

Thanks

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

 

Met vriendelijke groet,

 

David Shaw

Senior ICT consultant

 

Randstad Groep Nederland

Diemermere 15, 1112 TB Diemen

P.O. Box 12600, NL-1100 AP Amsterdam

david...@randstadgroep.nl

www.randstadgroep.nl