akka clustering ipaddress vs hostname issue

[Rohit Kumar Gupta]

Hi,

I am building my application based on akka distributed worker example on typesafe.  I am using play 2.3.1 (activator based) and akka 2.3.4. 

I am facing below error that keeps on popping on the console of worker process. I have a product cluster that has three processes - master, worker and frontend all of them part of a ProductCluster. I first tried configuring hostname (as I prefer it) in the akka config files. Worker process resolves the hostname as gdcacn-fase13 but other processes resolve it as gdcacn-fase13.ap.equinix.com so worker can't communicate with the cluster. It gets cutoff. 

Then I try configuring the ipaddress in the configuration file and I keep on getting the below error on the worker process. It is still not able to communicate with the cluster. 

Note:- Workers communicate with the cluster with ClusterClient but it also joins the same cluster as other processes. I want to keep everything as one cluster.

[ERROR] [07/25/2014 17:44:03.814] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2551-1] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%23-375727047#1209858025]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]

Any help will be very much appreciated.

Thanks & Regards,

Rohit

[Rohit Kumar Gupta]

Hi,

Akka 2.3.4 is having huge issue in resolving the hostname/ipaddress when used with clustering. Sometimes it resolves the as hostname and sometimes it resolves as ipaddress. This makes the addresses different and one part of the cluster can't communicate with the other part of the cluster. I keep on getting the below error. This is a serious issue and seems like a bug. 

Note:- I am only configuring the hostname everywhere but somehow it picks up the ipaddress.

Any information will be appreciated.

Thanks,

Rohit

[ERROR] [07/25/2014 20:08:31.349] [ProductCluster-akka.remote.default-remote-dispatcher-15] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                        

[ERROR] [07/25/2014 20:08:34.347] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                         

[ERROR] [07/25/2014 20:08:36.967] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40gdcacn-fase13.ap.equinix.com%3A2551-5] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%231908980676#798338406]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                                                                                                              

[ERROR] [07/25/2014 20:08:36.972] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40gdcacn-fase13.ap.equinix.com%3A2551-5] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%23-375727047#1209858025]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                                                                                                             

[ERROR] [07/25/2014 20:08:37.349] [ProductCluster-akka.remote.default-remote-dispatcher-15] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                        

[ERROR] [07/25/2014 20:08:40.347] [ProductCluster-akka.remote.default-remote-dispatcher-15] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                        

[ERROR] [07/25/2014 20:08:42.990] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40gdcacn-fase13.ap.equinix.com%3A2551-5] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%231908980676#798338406]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                                                                                                              

[ERROR] [07/25/2014 20:08:42.992] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40gdcacn-fase13.ap.equinix.com%3A2551-5] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%23-375727047#1209858025]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]                                                                                                                             

[ERROR] [07/25/2014 20:08:43.347] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]    

[Akka Team]

Hi Rohit,

Have you looked into the configuration setting akka.remote.netty.tcp.hostname ?

      # The hostname or ip to bind the remoting to,

      # InetAddress.getLocalHost.getHostAddress is used if empty
      hostname = ""

If getHostAddress does not return the address/hostname what other nodes refer this host then it will refuse connections.

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Roland Kuhn]

Hi Rohit,

Akka deliberately does no resolution etc. when interpreting the akka.remote.netty.tcp.hostname you configure: it will use exactly the given string when constructing the ActorSystem’s own address and it will bind to the address the O/S resolves for this string. This means that you must configure each system with the hostname by which all other nodes can reach it. I would recommend using FQDNs everywhere when configuring this manually and IP addresses when obtaining it from an external service (like the AWS API or similar).

When sending a message (including joining a cluster) you must use the exact same string for the hostname portion that the remote system has been configured with.

This rule might seem arbitrary or complex, but it is actually the simplest one, and it is the only possibility for Akka to stay clear of the myriad of different network administration schemes you might come up with.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Patrik Nordwall]

A side note...

25 jul 2014 kl. 11:54 skrev Rohit Kumar Gupta <rohitkum...@gmail.com>:

Hi,

I am building my application based on akka distributed worker example on typesafe.  I am using play 2.3.1 (activator based) and akka 2.3.4. 

I am facing below error that keeps on popping on the console of worker process. I have a product cluster that has three processes - master, worker and frontend all of them part of a ProductCluster. I first tried configuring hostname (as I prefer it) in the akka config files. Worker process resolves the hostname as gdcacn-fase13 but other processes resolve it as gdcacn-fase13.ap.equinix.com so worker can't communicate with the cluster. It gets cutoff. 

Then I try configuring the ipaddress in the configuration file and I keep on getting the below error on the worker process. It is still not able to communicate with the cluster. 

Note:- Workers communicate with the cluster with ClusterClient but it also joins the same cluster as other processes. I want to keep everything as one cluster.

ClusterClient is designed for usage from nodes not part of the cluster. If your client nodes have joined the cluster you can for example use DistributedPubSub instead.

/Patrik

[ERROR] [07/25/2014 17:44:03.814] [ProductCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://Product...@172.16.144.66:2552/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FProductCluster%40172.16.144.66%3A2551-1] dropping message [class com.equinix.fase.job.product.common.MasterWorkerProtocol$WorkIsReady] for non-local recipient [Actor[akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552/user/receptionist/akka%3A%2F%2FProductCluster%2Fuser%2Fworker%23-375727047#1209858025]] arriving at [akka.tcp://Product...@gdcacn-fase13.ap.equinix.com:2552] inbound addresses are [akka.tcp://Product...@172.16.144.66:2552]

Any help will be very much appreciated.

Thanks & Regards,

Rohit

--

[Rohit Kumar Gupta]

Thanks Roland and Patrik for your replies.

Regards,

Rohit