akka clusters on AWS ECS

[Eric Swenson]

Are there any good solutions for deploying akka clustered applications in docker containers running under AWS/ECS?  While there are proposed solutions for AWS/EC2 with autoscaling, none of them targets AWS/ECS.  In a nutshell, the issue is this:

If your cluster-nodes are implemented as docker containers, and formalized as ECS tasks that are members of an ECS service (in an ECS cluster), then when they are started and stopped (by ECS) in response to load, their IP addresses are impossible to predict. It is not possible to configure a fixed set of seed nodes at known IP addresses unless these seed nodes are managed independently of ECS automatic management. And, of course, having fixed, known seed nodes becomes problematic if/when these nodes fail.  

I could roll my own solution using a key/value store like redis, where whenever a node comes up, it first publishes its IP addresses (and port, if not common) to the key/value store, and then queries that store to configure its seed nodes. Care would have to be taken to clean up old entries when nodes fail, of course. But it seems like I shouldn't be the first person to run into akka-cluster issues on AWS/ECS and that other solutions might be out there. Unfortunately, I've found none.

Any ideas?

-- Eric

[Eric Swenson]

Probably a better roll-my-own solution would involve querying AWS/ECS dynamically to discover the IP addresses of the containers running in the cluster, excluding the IP address of node making the query, and setting the seed nodes to the remaining nodes. This way, the likely of getting the list of nodes out-of-sync with reality is minimal.  — Eric

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/MnxSZma4QJI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Roland Kuhn]

Yes, this sounds like a good strategy to me.

Regards,

Roland

You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Odd Möller]

Hi Eric, have you looked at https://github.com/hseeberger/constructr?

Greetings

Odd

[Eric Swenson]

Hi Odd,

Thanks. I've looked at constructr and see that it would work fine and is a general solution not tied to AWS/ECS. However, it does add more moving parts to a deployment and has the potential to get out of sync with the source of truth of the cluster definition -- ECS itself. I will, however, consider it, as there is some beauty in a general solution.  -- Eric

[Odd Möller]

Hi Eric

For an AWS specific solution perhaps Chris Loy's blog post on that subject can be of interest: http://chrisloy.net/2014/05/11/akka-cluster-ec2-autoscaling.html.

Greetings

Odd

[Eric Swenson]

Hello Odd,

Thanks. That article is what motivated me to write an AWS/ECS solution, since the details of cluster node discovery are different under AWS/ECS than under AWS/EC2/Autoscaling. 

Now mind you, it isn’t that I’m really looking for an AWS-specific solution — in fact the disadvantages of such a system are that it ties us to AWS. However, it does seem less error prone to consult the source of truth of cluster node addresses — AWS/ECS in this case rather than try to duplicate that state in another store/cache.  

— Eric

On Feb 2, 2016, at 10:30 AM, Odd Möller <odd.m...@gmail.com> wrote:

Hi Eric

For an AWS specific solution perhaps Chris Loy's blog post on that subject can be of interest: http://chrisloy.net/2014/05/11/akka-cluster-ec2-autoscaling.html.

Greetings

Odd

On 2 feb. 2016, at 19:01, Eric Swenson <er...@swenson.org> wrote:

Hi Odd,

Thanks. I've looked at constructr and see that it would work fine and is a general solution not tied to AWS/ECS. However, it does add more moving parts to a deployment and has the potential to get out of sync with the source of truth of the cluster definition -- ECS itself. I will, however, consider it, as there is some beauty in a general solution.  -- Eric

On Tuesday, February 2, 2016 at 1:15:17 AM UTC-8, Odd Möller wrote:

Hi Eric, have you looked at https://github.com/hseeberger/constructr?

Greetings

Odd

[David Santiago Osorio Vasquez]

You can share me the solution on AWS / ECS?