RabbitMQ connection refused

[Emran Belushi]

Hello Everyone,

Since this morning we are having problems with RabbitMQ. Yesterday I checked rabbitMQ via browser it was working fine.

This morning it has stopped working, the first time I went on it it showed:
 

undefined: There is no template at js/tmpl/login.ejs

Later it started showing

Now it is showing browser default error: ERR_Connection_Refused

I checked our website log files and it is logging "server:5672 unavailable. Error: No connection could be made because the target machine actively refused it"

I tried to run "rabbitmqctl status", it gave me response but for "{running_application, []}" the list was empty.

So I am not sure what is happening without making any amendments.

Any help will be appreciated.

Thank you


[Alvaro Videla]

Any info on the server logs?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emran Belushi]

Server log of RabbitMQ or website?

[Michael Klishin]

RabbitMQ 

[Emran Belushi]

There is only 1 line for today.

INFO REPORT 26-Nov
Stopping RabbitMQ


This is because I previously attempted to restart it, but it took forever so I pressed Ctrl+C to cancel.

The server is running, but there is no application running as when I run the status command, running applications is empty.

Thank you for taking out time.


[Emran Belushi]

There is another log file sasl.log.1, it had these details. The problem started between yesterday evening and this morning.

=CRASH REPORT==== 25-Nov-2015::19:48:56 ===

  crasher:

    initial call: gen:init_it/6

    pid: <0.102.0>

    registered_name: []

    exception exit: {unexpected_info,{mnesia_locker,xxx@dh2,granted}}

      in function  gen_server2:terminate/3

    ancestors: [worker_pool_sup,rabbit_sup,<0.91.0>]

    messages: []

    links: [<0.98.0>]

    dictionary: [{random_seed,{5627,15941,14233}},

                  {worker_pool_worker,true},

                  {fhc_age_tree,{0,nil}}]

    trap_exit: false

    status: running

    heap_size: 1598

    stack_size: 27

    reductions: 38856

  neighbours:

=SUPERVISOR REPORT==== 25-Nov-2015::19:48:56 ===

     Supervisor: {local,worker_pool_sup}

     Context:    child_terminated

     Reason:     {unexpected_info,{mnesia_locker,xxx@dh2,granted}}

     Offender:   [{pid,<0.102.0>},

                  {name,3},

                  {mfargs,{worker_pool_worker,start_link,[]}},

                  {restart_type,transient},

                  {shutdown,4294967295},

                  {child_type,worker}]

=CRASH REPORT==== 25-Nov-2015::19:48:56 ===

  crasher:

    initial call: gen:init_it/6

    pid: <0.100.0>

    registered_name: []

    exception exit: {unexpected_info,{mnesia_locker,xxx@dh2,granted}}

      in function  gen_server2:terminate/3

    ancestors: [worker_pool_sup,rabbit_sup,<0.91.0>]

    messages: []

    links: [<0.98.0>]

    dictionary: [{random_seed,{5945,24917,8746}},

                  {worker_pool_worker,true},

                  {fhc_age_tree,{0,nil}}]

    trap_exit: false

    status: running

[Emran Belushi]

rabbitmqctl status

Status of node xxx@dh1 ...

[{pid,16382},

 {running_applications,[]},

 {os,{unix,linux}},

 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:12:12] [async-threads:30] [kernel-poll:true]\n"},

 {memory,[{total,146300176},

          {connection_procs,585464},

          {queue_procs,635856},

          {plugins,0},

          {other_proc,13592688},

          {mnesia,100848},

          {mgmt_db,0},

          {msg_index,266992},

          {other_ets,1389256},

          {binary,16245216},

          {code,19874905},

          {atom,703377},

          {other_system,92905574}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,27010899968},

 {disk_free_limit,50000000},

 {disk_free,145986981888},

 {file_descriptors,[{total_limit,1048476},

                    {total_used,11},

                    {sockets_limit,943626},

                    {sockets_used,0}]},

 {processes,[{limit,1048576},{used,285}]},

 {run_queue,0},

 {uptime,12274619}]

...done.

[Michael Klishin]

On 26 November 2015 at 16:21:27, Emran Belushi (em...@nffd.co.uk) wrote:
> dictionary: [{random_seed,{5945,24917,8746}},
>
>
> {worker_pool_worker,true},

Not sure how many of those you have but this means a process in a pool used e.g. to
recover data from disk. I don't recall seeing this before.

Does this happen on every restart? 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Emran Belushi]

I didn't restart it, all of a sudden it has stopped working.

when I run stop or restart commands, it just hangs and I have to force it to stop with Ctrl + C. 

It has never happened before.

[Michael Klishin]

On 26 November 2015 at 17:07:04, Emran Belushi (em...@nffd.co.uk) wrote:
> when I run stop or restart commands, it just hangs and I have to
> force it to stop with Ctrl + C.

Can you post full SASL log? Is this with a single node or one of cluster nodes?

What version do you run? 

[Emran Belushi]

That is what the log had in it.

The version I am running is 3.3.5

[Michael Klishin]

On 26 November 2015 at 17:11:54, Emran Belushi (em...@nffd.co.uk) wrote:
> That is what the log had in it.
>
> The version I am running is 3.3.5

Unfortunately 3.3.x is not supported in any way.

I could find this error message in our old bug tracker. It ends up being
a standard library problem pre-17.x.

So try Erlang 17.5 and upgrade to 3.5.6  when you have a chance.

[Emran Belushi]

I'll upgrade when I have the chance, but is there a temporary solution for now?

and why there is no list of applications in running_applications when I run status command, doesn't that show the problem is due to applications not running ?

Thanks

[Michael Klishin]

On 26 November 2015 at 17:22:40, Emran Belushi (em...@nffd.co.uk) wrote:
> is there a temporary solution for now?

Try Erlang 17.5 or 18.0 (not 18.1, only 3.5.6 supports it)

> and why there is no list of applications in running_applications
> when I run status command, doesn't that show the problem is due
> to applications not running ?

There is a race condition in an embedded data store that causes it to terminate.
Things that depend on it cannot finish starting as a result, including RabbitMQ server and all of the
plugins. 

[Emran Belushi]

I rebooted the linux server and it started to work.