Cannot connect to RMQ - error,nxdomain

[Lukáš Krahulec]

Hello,

I have problem, which happen very irregularly, sometimes it is not possible to connect to RMQ. This errors starts when we upgrade from version 3.6.10 to 3.7.0 (now we are using 3.7.4). 

I found on this problem that problem is with resolving hostname, but we dont use clustering, no VPN and i added hostname into "hosts" file (RMQ is running on Windows server).

It is possible to configure RMQ to completly disable cluster or do not resolve hostname or is there any other solution for this problem.

In crash.log I see:

2018-04-16 22:41:16 =CRASH REPORT====

  crasher:

    initial call: rabbit_reader:init/4

    pid: <0.26542.3>

    registered_name: []

    exception error: {{badmatch,{error,nxdomain}},[{rabbit_nodes,cluster_name_default,0,[{file,"src/rabbit_nodes.erl"},{line,77}]},{rabbit_nodes,cluster_name,0,[{file,"src/rabbit_nodes.erl"},{line,72}]},{rabbit_reader,server_properties,1,[{file,"src/rabbit_reader.erl"},{line,251}]},{rabbit_reader,start_connection,3,[{file,"src/rabbit_reader.erl"},{line,1060}]},{rabbit_reader,handle_input,3,[{file,"src/rabbit_reader.erl"},{line,1010}]},{rabbit_reader,recvloop,4,[{file,"src/rabbit_reader.erl"},{line,461}]},{rabbit_reader,run,1,[{file,"src/rabbit_reader.erl"},{line,443}]},{rabbit_reader,start_connection,4,[{file,"src/rabbit_reader.erl"},{line,362}]}]}

    ancestors: [<0.26540.3>,<0.1193.0>,<0.1192.0>,<0.1191.0>,rabbit_sup,<0.255.0>]

    message_queue_len: 1

    messages: [{'EXIT',#Port<0.1226099>,normal}]

    links: [<0.26540.3>]

    dictionary: [{process_name,{rabbit_reader,<<"192.168.110.12:54598 -> 192.168.110.12:5672">>}}]

    trap_exit: true

    status: running

    heap_size: 987

    stack_size: 27

    reductions: 2887

  neighbours:

2018-04-16 22:41:16 =SUPERVISOR REPORT====

     Supervisor: {<0.26540.3>,rabbit_connection_sup}

     Context:    child_terminated

     Reason:     {{badmatch,{error,nxdomain}},[{rabbit_nodes,cluster_name_default,0,[{file,"src/rabbit_nodes.erl"},{line,77}]},{rabbit_nodes,cluster_name,0,[{file,"src/rabbit_nodes.erl"},{line,72}]},{rabbit_reader,server_properties,1,[{file,"src/rabbit_reader.erl"},{line,251}]},{rabbit_reader,start_connection,3,[{file,"src/rabbit_reader.erl"},{line,1060}]},{rabbit_reader,handle_input,3,[{file,"src/rabbit_reader.erl"},{line,1010}]},{rabbit_reader,recvloop,4,[{file,"src/rabbit_reader.erl"},{line,461}]},{rabbit_reader,run,1,[{file,"src/rabbit_reader.erl"},{line,443}]},{rabbit_reader,start_connection,4,[{file,"src/rabbit_reader.erl"},{line,362}]}]}

     Offender:   [{pid,<0.26542.3>},{name,reader},{mfargs,{rabbit_reader,start_link,[<0.26541.3>,{acceptor,{0,0,0,0},5672},#Port<0.1226099>]}},{restart_type,intrinsic},{shutdown,30000},{child_type,worker}] 

Thanks for help.

Lukas

[Luke Bakken]

Hi Lukas,

What version of Erlang are you using?

When you run rabbitmqctl status, the first output line will be:

Status of node rabbit@HOST...

In your case, does HOST match the output of the hostname command, and does it match the entry you put in the hosts file?

Finally, if you go to the "Advanced System Settings" window, then "Computer Name", does "Computer Name" and "Full Computer Name" match the host name you expect?

Thanks,

Luke

[Lukáš Krahulec]

Erlang is 20.3

Hostname from "rabbitmqctl status", entry in hosts and Computer and Full computer name are same.

But "rabbitmqctl status" shows error:

Status of node rabbit@CP-MASTER ...

Error: unable to perform an operation on node 'rabbit@CP-MASTER'. Please se

e diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection o

r firewall issues)

 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang

 cookie not matching that of the server)

 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/document

ation.html to learn more

 * Consult server logs on node rabbit@CP-MASTER

DIAGNOSTICS

===========

attempted to contact: ['rabbit@CP-MASTER']

rabbit@CP-MASTER:

  * connected to epmd (port 4369) on CP-MASTER

  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traff

ic

  * TCP connection succeeded but Erlang distribution failed

  * Authentication failed (rejected by the remote node), please check the Erlang

 cookie

Current node details:

 * node name: 'rabbitmqcli42@CP-MASTER'

 * effective user's home directory: C:\Users\Administrator

 * Erlang cookie hash: CKJj3JM6RHTjIeGCl6kIYw==

Dne středa 18. dubna 2018 17:00:42 UTC+2 Luke Bakken napsal(a):

[Luke Bakken]

Hi Lukáš,

You should copy the Erlang cookie from:

C:\WINDOWS\system32\config\systemprofile\.erlang.cookie

to:

C:\Users\Administrator\.erlang.cookie

rabbitmqctl status should work fine after that.

Thanks,
Luke

[Lukáš Krahulec]

Now status is without errors. But any more ideas with error,nxdomain problem or copy cookie should help with this problem too ? It is possible to disable resolving hostname? I don't need it and I don't understand why RMQ needs to resolve hostname.

Thank you

Lukas
 
Dne středa 18. dubna 2018 17:55:34 UTC+2 Luke Bakken napsal(a):

[Michael Klishin]

It uses local hostname in default cluster name:

https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_nodes.erl#L72

and the default value is computed unconditionally.

It is possible to customise hostname resolution in the runtime [1] by providing a local file and a VM argument that points at the file.

However, I personally haven't tried the "domain" option which seems to be what you need in this case.

I'd also add that a system where it's not possible to resolve current hostname is likely to fail in confusing ways in many other

areas. A lot of software just assumes that there *is* a current hostname. Consider fixing that problem instead of finding workarounds.

1. http://erlang.org/doc/apps/erts/inet_cfg.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Luke Bakken]

Hi everyone,

I've opened the following pull request which will make getting the host name a little more forgiving -

https://github.com/rabbitmq/rabbitmq-server/pull/1580

Since this name is used for the cluster name it's OK to default to the value returned by inet:gethostname/0, I think.

Lukáš - you can give this a try in a future 3.8.0 alpha build, or wait for the next release.

Thanks,

Luke

[Lukáš Krahulec]

Thanks for answer.

I thought that erlang use hosts file (in windows).

I have:

192.168.1.12 CP-MASTER

And if I resolve in werl console

inet:gethostbyname('CP-MASTER')

it return 192.168.1.12. But when I try to change IP in host to for example 192.168.1.55 resolving in werl still return 192.168.1.12 (I tried flush dns)

But when I try to add some test entry into host file

192.168.1.13 test

and try to resolve inet:gethostbyname('test'), it return right IP.

So I dont know where host name CP-MASTER is resolved, it look like that host file is not use when resolving hostname.

Dne čtvrtek 19. dubna 2018 18:48:26 UTC+2 Michael Klishin napsal(a):

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Erlang does use /etc/hosts (among other things) like any other processes unless it is instructed

to do otherwise via an inetrc file [1].

Luke has submitted a PR that should make RabbitMQ more forgiving, it is undergoing QA.

1. http://erlang.org/doc/apps/erts/inet_cfg.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.