* unable to connect to epmd (port 4369) on ip-10-0-1-236: timeout (timed out)

[Gaurav Kalmady]

Hi,
  I am running Rabbitmq v3.3.5 on an EC2 Node
I get the following error
[root@ip-10-0-1-107 ~]# rabbitmqctl -n pythia@ip-10-0-1-236 status
Status of node 'pythia@ip-10-0-1-236' ...
Error: unable to connect to node 'pythia@ip-10-0-1-236': nodedown

 I intend to cluster the rabbits on 236 and 107. They both are running rabbitmq v 3.3.5. 

The status is as follows

[root@ip-10-0-1-236 ~]# rabbitmqctl status
Status of node 'pythia@ip-10-0-1-236' ...
[{pid,27952},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},
      {webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},
      {rabbit,"RabbitMQ","3.3.5"},
      {os_mon,"CPO  CXC 138 46","2.2.14"},
      {inets,"INETS  CXC 138 49","5.9.7"},
      {mnesia,"MNESIA  CXC 138 12","4.11"},
      {amqp_client,"RabbitMQ AMQP Client","3.3.5"},
      {xmerl,"XML parser","1.3.5"},
      {sasl,"SASL  CXC 138 11","2.3.4"},
      {stdlib,"ERTS  CXC 138 10","1.19.4"},
      {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,
     [{total,40153280},
      {connection_procs,8416},
      {queue_procs,5408},
      {plugins,290824},
      {other_proc,14066560},
      {mnesia,71912},
      {mgmt_db,63960},
      {msg_index,22392},
      {other_ets,1087928},
      {binary,124152},
      {code,19484890},
      {atom,703377},
      {other_system,4223461}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,3142454476},
 {disk_free_limit,50000000},
 {disk_free,48830353408},
 {file_descriptors,
     [{total_limit,924},{total_used,3},{sockets_limit,829},{sockets_used,1}]},
 {processes,[{limit,1048576},{used,180}]},
 {run_queue,0},
 {uptime,10}]
...done.

I try to access this machine from my other(cluster sibling) rabbit machine and I get the aforementioned error.

[root@ip-10-0-1-107 ~]# rabbitmqctl -n pythia@ip-10-0-1-236 status
Status of node 'pythia@ip-10-0-1-236' ...
Error: unable to connect to node 'pythia@ip-10-0-1-236': nodedown


The hosts file is ok on both machines

[root@ip-10-0-1-236 ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost ip-10-0-1-236
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


[root@ip-10-0-1-107 ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost ip-10-0-1-107
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.1.236  ip-10-0-1-236

They share Erlang cookies

[root@ip-10-0-1-236 ~]# cat /var/lib/rabbitmq/.erlang.cookie
LHUVLCQNOPJUGDWXYNDR[root@ip-10-0-1-236 ~]#

[root@ip-10-0-1-107 ~]# cat /var/lib/rabbitmq/.erlang.cookie
LHUVLCQNOPJUGDWXYNDR[root@ip-10-0-1-107 ~]#

The ports are open on ip-10-0-1-236
[root@ip-10-0-1-236 ~]# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:epmd
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:amqp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:15672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:25672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:epmd
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:amqp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:15672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:25672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:epmd
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:amqp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:15672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:25672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:atmtcp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:epmd
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:amqp
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:15672
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:25672
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


epmd seems to be running and listening

ps aux | grep epmd
rabbitmq 22584  0.0  0.0  10956   612 ?        S    Nov20   0:02 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
root     28273  0.0  0.0   6392   688 pts/0    S+   21:42   0:00 grep epmd


[root@ip-10-0-1-236 ~]# netstat -uptan | grep epmd
tcp        0      0 0.0.0.0:4369                0.0.0.0:*                   LISTEN      22584/epmd
tcp        0      0 127.0.0.1:4369              127.0.0.1:33211             ESTABLISHED 2


So now what remains? Why can I not connect this rabbit instance?

[Michael Klishin]

On 25 November 2014 at 00:44:39, Gaurav Kalmady (gaurav...@gmail.com) wrote:
> > The hosts file is ok on both machines
>
> [root@ip-10-0-1-236 ~]# cat /etc/hosts
> 127.0.0.1 localhost.localdomain localhost ip-10-0-1-236
> ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
>
>
> [root@ip-10-0-1-107 ~]# cat /etc/hosts
> 127.0.0.1 localhost.localdomain localhost ip-10-0-1-107
> ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
> 10.0.1.236 ip-10-0-1-236

Note that all nodes need to resolve all others. 236 currently don't seem to have
an entry for 107 (or was it omitted?)

The above INPUT chain suggests port 4369 is blocked. Try connecting to
it with telnet, for example. 

See Firewalled Nodes on http://www.rabbitmq.com/ec2.html.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Gaurav Kalmady]

Thanks MK,
   I found out that the port was blocked via security group in EC2. I have opened the port in the EC2 security group. However I thought the IP table reflected that the port is open here. Does this not mean that?

ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:epmd

Also after adding 4369 to the security group in EC2, and restarting rabbit, I continue to get the same error.

Thanks
Gaurav

[Michael Klishin]

On 25 November 2014 at 01:12:25, Gaurav Kalmady (gaurav...@gmail.com) wrote:
> I found out that the port was blocked via security group in EC2.
> I have opened the port in the EC2 security group. However I thought
> the IP table reflected that the port is open here. Does this not
> mean that?
> ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:epmd

This means that epmd traffic will be accepted, yes.

> Also after adding 4369 to the security group in EC2, and restarting
> rabbit, I continue to get the same error.

RabbitMQ doesn't need to be restarted after opening the ports.

This may be another issue. Can you confirm that telnet connections to 236
on port 4369 succeed (epmd may quickly disconnect you because you probably won't
send it what it expects but you should be able to connect) ?

[Gaurav Kalmady]

Hi MK,
  Thanks for your quick responses. Further investigation showed that the ports 4369 and 25672 were closed in the EC2 security group but open via IPtables. Opening both solved the issue.
Thanks
Gaurav

[Vinod Yadav]

Hi Gaurav,

Can you provide the steps how you resolved this error I am stuck with the same error?

Thanks 

vinod