Akka Cluster : Association with remote system has failed

[Manoj Santhakumaran]

Hi,

I'm working on a Play/Akka application with cluster system of two nodes, and each node is deployed in docker container in two different hosts.

My log files are filled up with following message, which comes every 5/10 milli seconds.

WARN  | 2016-06-21 01:00:03 | [ClusterSystem-akka.actor.default-dispatcher-21] ? - [?] - Association with remote system [akka.tcp://ClusterSystem@hostname2:54797] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://ClusterSystem@hostname2:54797]] Caused by: [Connection refused: hostname2/xxx.xxx.xxx.xxx:54797]

Can anyone help me to understand and fix this issue?

My application.conf is as follows:

akka {
    actor.provider = "akka.cluster.ClusterActorRefProvider"
    ...
    cluster {
        seed-nodes = [
            "akka.tcp://ClusterSystem@hostname1:2551",
            "akka.tcp://ClusterSystem@hostname2:2552"]
    }
    ...
}

And I'm passing port and host name as JVM args for both of the instances (Eg: -Dakka.remote.netty.tcp.port=2551/2552 -Dakka.remote.netty.tcp.hostname=hostname1/hostname2)

I verified the network connectivity between these two hosts using "ping" command inside docker container. There was no packet loss and looked fine.

Thank you very much.
Manoj

[Manoj Santhakumaran]

Today's update:

I tried setting the bind-hostname also:

akka.remote.netty.tcp.bind-hostname = 0.0.0.0

But no luck !!

I verified the connectivity between two docker containers, from inside container.

telnet hostname1 2551 (From hostname2 works)
telnet hostname2 2552 (From hostname1 works)
host hostname1 (From hostname2 works)
host hostname2 (From hostname1 works)

I'm using Akka 2.4.4 version.

Appreciate if anyone have any idea/pointers what could be going wrong here?

Thank you very much.
Manoj

[Patrik Nordwall]

Try to run with the remote debug logging enabled. Described in logging section of documentation.

I'm not sure where the port number 54947 comes from but we might be able to see that if we look at full debug logs.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Manoj Santhakumaran]

Hi Patrik,

Thank you very much for your reply.

I enabled the Akka remote logging and ran both of my nodes. Attached here the logs for your perusal. Any findings/pointers are highly appreciated.

On Tuesday, 21 June 2016 12:36:10 UTC-7, Patrik Nordwall wrote:

Try to run with the remote debug logging enabled. Described in logging section of documentation.

I'm not sure where the port number 54947 comes from but we might be able to see that if we look at full debug logs.

Today I got a different port number at this line of the log file:

WARN  | 2016-06-22 20:00:03 | [ClusterSystem-akka.actor.default-dispatcher-17] ? - [sourceThread=ClusterSystem-akka.remote.default-remote-dispatcher-8, sourceActorSystem=ClusterSystem, akkaSource=akka.tcp://ClusterSystem@hostname2:43209/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40hostname1%3A42282-207, akkaTimestamp=20:00:03.446UTC] - Association with remote system [akka.tcp://ClusterSystem@hostname1:42282] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://ClusterSystem@hostname1:42282]] Caused by: [Connection refused: hostname1/xxx.xxx.xxx.xxx:42282]

[Mark Hatton]

Hi Manoj,

From the logs it appears that you have a 4-member cluster, but I imagine you are attempting to construct a 2-member cluster.

Is it possible that you are launching 2 instances of your app on each host?  Or perhaps more likely you have 2 actor systems within each app?

And I would guess the appearance of the "random" port numbers is due to the unintended cluster members not having a configured port number (i.e. it is set to zero).  This will cause a random available port to be allocated and used by Akka cluster.  From the docs (http://doc.akka.io/docs/akka/2.4.7/general/configuration.html#akka-remote):

Default is 2552 (AKKA), use 0 if you want a random available port

Mark

[Siva Kommuri]

Are the following values set for hostname1:

message.http-bindport=2551
akka.remote.netty.tcp.hostname=hostname1

And likewise for the second one as well?

Best Wishes,

Siva Kommuri

[Manoj Santhakumaran]

Hi Mark,

 

Thank you very much for your reply.

 

To answer your questions, yes I’m running two actor systems in each of the nodes. I have configured Play’s internal actor system with a role “backend” for starting cluster client manager and cluster client proxy, with port 2551. And I’m starting a new actor system with role “frontend” to run work producer and health check verification of external resources, with a port number set as 0.

 

Honestly this is not my idea, but found in following GIT Hub location.

 

https://github.com/typesafehub/activator-akka-distributed-workers-java

 

Appreciate any further comments/solutions on this.

 

Thanks and regards,

Manoj

[Manoj Santhakumaran]

Hi Siva,

 

Thank you very much for your reply.

 

I have set the following configurations in my nodes.

 

In “hostname1”, I’m setting JVM args “-Dakka.remote.netty.tcp.port=2551” and “-Dakka.remote.netty.tcp.hostname=hostname1”. And the “application.conf” is having following configuration as well.

akka.remote.netty.tcp.bind-hostname = 0.0.0.0

 

In “hostname2”, I have JVM args “-Dakka.remote.netty.tcp.port=2552” and “-Dakka.remote.netty.tcp.hostname=hostname2”. And the “application.conf” is having following configuration.

akka.remote.netty.tcp.bind-hostname = 0.0.0.0

 

I couldn’t find a configuration “message.http-bindport” anywhere, my bad. Please give any pointers on this configuration.

 

Appreciate any further comments.

 

Thanks and regards,

Manoj

[Siva Kommuri]

Hi Manoj,

Sorry, please ignore the `message.http-bindport`. The interesting args as you have mentioned are correct:

```

akka.remote.netty.tcp.port=2551

```

[Siva Kommuri]

Hi Manoj,

Sorry, please disregard message.http-bindport.

You have already configured the interesting parameters:

akka.remote.netty.tcp.port=2551
akka.remote.netty.tcp.hostname=hostname1

And then when configuring the seed nodes, it would be akka.cluster.seed-nodes as you have described in your earlier email.

So, are you still having issues?

[Mark Hatton]

Hi Manoj,

With regards to the "frontend" actor systems, have you considered that firewall rules (iptables, etc.) may be denying access to the random port from other hosts.  Why not use a fixed port instead?  Or, like Siva suggests, is it possible that the bind-hostname is not being configured for these actor systems?

But I'm wondering if you really want the "frontend" actor systems to be members of the cluster at all?  Have you considered simply omitting the cluster actor provider from your config, i.e. remove this line:

akka.actor.provider = "akka.cluster.ClusterActorRefProvider"

Regards,

Mark

[Manoj Santhakumaran]

Hi Siva,

Thank you for your reply. Yes I still have the issue.
But I got some suggestions from Mark and started working on it.

Thanks and regards,
Manoj

[Manoj Santhakumaran]

Hi Mark,

Thank you very much for your suggestion. Your point is well taken. I don't need my "frontend" actor system to be a cluster member.

I started correcting this in my code by providing a different configuration for "frontend" system.

Will post you my results soon I finish the changes.

Thanks and regards,
Manoj