Rabbitmq dying during startup, “epmd error for host”

[gw...@helloglobo.com]

Hello,

Rabbitmq keeps dying within 2 minutes of startup, and fails with the error:

ERROR: epmd error for host localhost: timeout (timed out)

The /etc/hosts file shows:

127.0.0.1   localhost Users-MacBook-Pro Users-MacBook-Pro.local

255.255.255.255 broadcasthost

::1             localhost

The /etc/hosts file has undergone a few iterations, it started out with just the

 127.0.0.1 locahost itself, but I added more 127.0.0.1 lines to cover any hostna

me that could come up. After my boss looked at his, we put them all on the same

line as it now stands to mimic his.

I have tried uninstalling and reinstalling Erlang (v20.3.8.3) and Rabbitmq via homebrew, and still no connection. This is also regardless of what wifi I am on, or if I'm disconnected from wifi. Other commands I have tried:

$ echo $HOSTNAME

localhost

$ hostname

localhost

$ ifconfig lo0

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

    options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>

    inet 127.0.0.1 netmask 0xff000000 

    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1

    nd6 options=201<PERFORMNUD,DAD>

$ lsof -p 15279

COMMAND   PID USER   FD   TYPE             DEVICE   SIZE/OFF    NODE NAME

epmd    15279 user  cwd    DIR                1,5        960       2 /

epmd    15279 user  txt    REG                1,5      46756 4852991 /usr/local/Cellar/erlang@20/20.3.8.3/lib/erlang/erts-9.3.3.2/bin/epmd

epmd    15279 user  txt    REG                1,5     841456 2333502 /usr/lib/dyld

epmd    15279 user  txt    REG                1,5 1170673664 3866438 /private/var/db/dyld/dyld_shared_cache_x86_64h

epmd    15279 user    0r   CHR                3,2        0t0     311 /dev/null

epmd    15279 user    1w   CHR                3,2        0t0     311 /dev/null

epmd    15279 user    2w   CHR                3,2        0t0     311 /dev/null

epmd    15279 user    3u  IPv4 0x91f821cb489372bb        0t0     TCP *:epmd (LISTEN)

epmd    15279 user    4u  IPv6 0x91f821cb3ac82ed3        0t0     TCP *:epmd (LISTEN)

epmd    15279 user    5u  IPv4 0x91f821cb4a2f057b        0t0     TCP localhost:epmd->localhost:55223 (ESTABLISHED)

epmd    15279 user    6u  IPv4 0x91f821cb3a64dc1b        0t0     TCP localhost:epmd->localhost:55470 (ESTABLISHED)

epmd    15279 user    7u  IPv4 0x91f821cb4919995b        0t0     TCP localhost:epmd->localhost:55974 (ESTABLISHED)

$ lsof -nP -i4TCP:4369 | grep LISTEN

epmd    15279 user    3u  IPv4 0x91f821cb489372bb      0t0  TCP *:4369 (LISTEN)

$ telnet localhost 4369

Trying ::1...

Connected to localhost.

Escape character is '^]'.

I'm not sure what else to try here, and am hoping someone has an idea.

Thank you, 

Gwen.

[Michael Klishin]

Hi Gwen,

All previous discussions of this that show up in a quick search [1][2][3] come down to either hostname

resolution (for both "localhost" but also the default hostname) or epmd binding to an unexpected interface.

The output of lsof, the telnet connection and your example /etc/hosts all look reasonable. Are you on VPN by any chance?

The 2 minute timeout is a strong indication of a hostname resolution that does not use the mechanism the user expects.

We see this on MacOS from time to time, in particular on corporate VPNs.

Also, I don't see anything suspicious in the Homebrew formula but it would be much easier to reason about things

if you could use the generic UNIX package (how Erlang is installed should not matter).

You can also kill `epmd` with `epmd -kill`, list known nodes with `epmd -names` and run it in foreground with debug logging `epmd -d`.

If `epmd -kill` doesn't help the easiest option is to `kill -9` it and let RabbitMQ start it on the next boot.

HTH.

1. https://stackoverflow.com/questions/24797947/os-x-and-rabbitmq-error-epmd-error-for-host-xxx-address-cannot-connect-to-ho

2. https://groups.google.com/d/msg/rabbitmq-users/gkOms0lGQ0E/GcUlh6MQJRIJ

3. https://gist.github.com/jch/2522701

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Gwen Schmidt]

Thank you very much for the help! Before I go to installing rabbitmq from UNIX (I have reasons for not wanting to go that way if I can avoid it), I did run epmd-d. Here’s what I got:

localhost:bin gwen$ ./epmd -d

epmd: Wed Aug  1 15:28:34 2018: epmd running - daemon = 0

epmd: Wed Aug  1 15:28:40 2018: ** got ALIVE2_REQ

epmd: Wed Aug  1 15:28:40 2018: registering 'rabbitmqprelaunch34325:2', port 52352

epmd: Wed Aug  1 15:28:40 2018: type 77 proto 0 highvsn 5 lowvsn 5

epmd: Wed Aug  1 15:28:40 2018: ** sent ALIVE2_RESP for "rabbitmqprelaunch34325"

epmd: Wed Aug  1 15:28:40 2018: ** got NAMES_REQ

epmd: Wed Aug  1 15:28:40 2018: ** sent NAMES_RESP

epmd: Wed Aug  1 15:28:40 2018: unregistering 'rabbitmqprelaunch34325:2', port 52352

epmd: Wed Aug  1 15:28:40 2018: ** got ALIVE2_REQ

epmd: Wed Aug  1 15:28:40 2018: registering 'rabbit:2', port 25672

epmd: Wed Aug  1 15:28:40 2018: type 77 proto 0 highvsn 5 lowvsn 5

epmd: Wed Aug  1 15:28:40 2018: ** sent ALIVE2_RESP for "rabbit"

epmd: Wed Aug  1 15:28:42 2018: unregistering 'rabbit:2', port 25672

So… it looks like rabbit is indeed getting to epmd just fine? The unregistered line is when rabbitmq dies off quietly.

How can I get rabbitmq to run in a better debug mode so that I can figure out what it’s doing? 

Thank you,

Gwen.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Luke Bakken]

Hi Gwen,

What is in your RabbitMQ logs? Can you attach an archive of the entire log file? Can you search for a file named erl_crash.dump? I'm not sure where homebrew installs RabbitMQ by default, so you may have to use find to search for that file.

Thanks,

Luke

[Gwen Schmidt]

Hello,

Well, then. Apparently the problem was that it couldn’t log.

So, oddly, the erl_crash.dump log was not generating last week when I looked for it (ran a find across the entire system even), but to be fair I have reinstalled rabbitmq a few times since. In the process, I was trying to get rabbitmq to log somewhere, and apparently set it to a bad path in the meantime. I figured this out from looking at erl_crash.dump now….

Thanks everyone! It’s alive!

Gwen.

[Luke Bakken]

Hi Gwen,

Would you mind pasting the content from erl_crash.dump that allowed you to figure out the issue? I'm sure it will assist someone else on this list in the future.

Thanks -

Luke

[Gwen Schmidt]

Hello,

The pertinent line was:

Slogan: init terminating in do_boot ({error,{cannot_log_to_file,/var/log/rabbitmq,enoent}})

It was the first non-timestamp line. 

Thanks!

Gwen.

On Aug 1, 2018, at 4:27 PM, Luke Bakken <lba...@pivotal.io> wrote:

Hi Gwen,

Would you mind pasting the content from erl_crash.dump that allowed you to figure out the issue? I'm sure it will assist someone else on this list in the future.

Thanks -

Luke

On Wednesday, August 1, 2018 at 1:25:17 PM UTC-7, Gwen Schmidt wrote:

Hello,

Well, then. Apparently the problem was that it couldn’t log.

So, oddly, the erl_crash.dump log was not generating last week when I looked for it (ran a find across the entire system even), but to be fair I have reinstalled rabbitmq a few times since. In the process, I was trying to get rabbitmq to log somewhere, and apparently set it to a bad path in the meantime. I figured this out from looking at erl_crash.dump now….

Thanks everyone! It’s alive!

Gwen.

On Aug 1, 2018, at 4:12 PM, Luke Bakken <lba...@pivotal.io> wrote:

Hi Gwen,

What is in your RabbitMQ logs? Can you attach an archive of the entire log file? Can you search for a file named erl_crash.dump? I'm not sure where homebrew installs RabbitMQ by default, so you may have to use find to search for that file.