[DesignProposal] AKKA cluster in Docker Swarm Environment

[Evgeny Shepelyuk]

Hello

Preface

Couple day ago I've asked a question here about possible design for setting AKKA cluster in Docker swarm environment.

The main open questions for us is how to discover seed nodes when new container is starting or service is scaled, because Docker assign random internal IPs for each container.

As far as I've seen from the reply and after skimming over similar Google results - current approach is to use some service registry like Consul / Etcd / Zookeeper and take seed nodes IP from there.

Since Docker Swarm provides own means for service discovery, we'd like to design approach that will not involve 3rd party service discovery like Consul.

After some thinking about it I'd like to share our vision and get some feedback on it.

A Docker Swarm background first. 

Cluster consists of manager and workers nodes, where managers maintain consistent internal state using raft implementation of the entire swarm and all the services running on it.

And workers are used for running container and workers nodes discover each other using gossip.

There also two types of service

• replicated services - the swarm manager distributes a specific number of replica tasks among the nodes based upon the scale you set in the desired state.
• global services -  the swarm runs one task for the service on every available node in the cluster.

Docker Swarm uses service discovery / load balancing based on symbolic service name.

Design

1. Let's assume we have a stateful AKKA service named wallet that is going to be sharded, so we'd like to leverage AKKA cluster functionality.
   
2. The service is run inside Docker container in Docker Swarm, AKKA service is a Docker Swarm service, i.e can be scaled over multiple nodes, service can be discovered within cluster by its name wallet.
   
3. To make seed nodes discovery possible we propose to introduce new dummy service wallet-seed and make this service global, i.e. docker swarm will run instance of this service on each node.
   
4. Additionally we can restrict deployment of wallet-seed only to manager nodes.
   
5. Doing as explained above allows us to ensure that wallet-seed is strongly available and run on each of manager of Docker Swarm cluster.
   
6. So when starting up, wallet service will use symbolic name wallet-seed as a seed node address to join to.
   
7. wallet may be launched on any Swarm node, either worker or manager.
8. Due to Docker Swarm internal load balancing, wallet-seed will be resolved to one of the available container instances.
   

Please keep in mind that this is a only initial proposal and the main goal is just to implement smth without using consul / etcd.
We will be appreciated for any feedback.

[Rafał Krzewski]

Evgeny,

The role of etcd / consul / zookeeper in the booting up an Akka cluster is to provide a distributed lock rather than service discovery. When you boot up a number of uniform nodes they need to decide among themselves which one will act as a seed of Akka cluster. The other nodes then join the seed. An external service that provides consistency guarantees prevents the cluster from brain-splitting at the very outset. I'm not familiar with docker-swarm feature set, but unless it provides a strongly consistent KV store or other means of distributed coordination, this is not going to fly.

Cheers,

Rafał

[Evgeny Shepelyuk]

Hello

Do you mean that in AKKA cluster there's only one seed node ? 

I thought AKKA cluster may have several seed node, so new instances can connect to any of them.

AKKA documentations show example with multiple seed nodes.

середа, 28 грудня 2016 р. 12:48:33 UTC+2 користувач Rafał Krzewski написав:

[Rafał Krzewski]

If you can set the IPs of your seed node(s) upfront and put them into configuration of every node all is fine and well. Unfortunately this is not what happens in a dynamic environment, hence the need to elect the seed node before Akka cluster boots up.

Cheers,

Rafał

[Evgeny Shepelyuk]

Hello,

In Docker Swarm when service is created, it's assigned to Virtual IP. I.e. when I create several instances of wallet-seed they will get an single IP (10.0.0.2, for example), while each of container will have own IP (10.0.0.5, 10.0.0.6, 10.0.0.7 and so on)

When wallet-seed address is resolved within Swarm cluster - it will initially resolves to service VIP (10.0.0.2) and then to the IP of particular container in round robin fashion.

So if I put wallet-seed  as a seed node address in my wallet service config, then 1st instance could resolve it to 10.0.0.5, second instance can resolve to 10.0.0.6 etc.

As far as I understand this will lead to cluster partitioning, right ?

середа, 28 грудня 2016 р. 13:06:22 UTC+2 користувач Rafał Krzewski написав:

[Rafał Krzewski]

W dniu środa, 28 grudnia 2016 13:30:23 UTC+1 użytkownik Evgeny Shepelyuk napisał:

Hello,

In Docker Swarm when service is created, it's assigned to Virtual IP. I.e. when I create several instances of wallet-seed they will get an single IP (10.0.0.2, for example), while each of container will have own IP (10.0.0.5, 10.0.0.6, 10.0.0.7 and so on)

When wallet-seed address is resolved within Swarm cluster - it will initially resolves to service VIP (10.0.0.2) and then to the IP of particular container in round robin fashion.

So if I put wallet-seed  as a seed node address in my wallet service config, then 1st instance could resolve it to 10.0.0.5, second instance can resolve to 10.0.0.6 etc.

As far as I understand this will lead to cluster partitioning, right ?

No, I meant something else. First of all, Akka cluster is implemented on top of Akka remoting. And for Akka remoting to work, each node needs to know it's self address, which is the IP the other nodes will use to connect to it. The node address becomes the part of remote ActorRefs. Any incoming message that contains different address than the node's self address is dropped. This is a good thing, because you could have actors with the same local paths on different node clusters and the results of delivering message to a wrong instance could potentially be disastrous. Because of this, even if you create wallet-seed service with virtual IP 10.0.0.2 you cannot use this IP as the address of the seed, because the containers backing the service will expect to receive messages addressed to 10.0.0.5, 10.0.0.6 and so on and will discard messages addressed to 10.0.0.2.

You could use docker-swarm service in a different way: to advertise the proper IP address of the cluster seed, but you cannot use Akka remoting protocol for that, for reason given above. Suppose you implement it the following way: define a HTTP endpoint, say at port 5000 that will respond to GET /seeds with a text/plain message containing IP address of cluster seed. Each node upon startup will do the following:

- attempt a GET wallet-seed:5000/seed 

- if it succeeds, join the cluster at received IP

- if it fails, this node is the first one and it should:

  - bootstrap the cluster by joining itself

  - publish a HTTP service at port 5000 advertising it's own address

The definition of wallet-seed service should include a health check condition so that only containers publishing the HTTP at port 5000 would be act as it's backend

That would work, but has a glaring problem: there is a race condition between the seed nodes. More than one container can "elect itself" as the seed. Then the other nodes would access different seeds through wallet-seed in round-robin fashion and join their respective clusters.

Using a consistent store like consul or etcd provides a solution to the race condition but also solves a second problem: it can be used to advertise seed address without crafting a custom service.

Cheers,

Rafał

[Evgeny Shepelyuk]

Hello

Thanks for detailed explanation. I assume we will proceed to distributed lock as you've suggested.

One last question, do you think Zookeeper can play this role well ? Or we should consider only Consul / Etcd ?
We're using Kafka internally, so we already having Zookeeper in place. 

середа, 28 грудня 2016 р. 15:06:08 UTC+2 користувач Rafał Krzewski написав:

[Rafał Krzewski]

Hi,
You are welcome!
You can use https://github.com/hseeberger/constructr together with https://github.com/typesafehub/constructr-zookeeper to initialize Akka cluster using Zookeeper.

Cheers,
Rafał

[Evgeny Shepelyuk]

Hello
Successfully implemented cluster joining using ConstructR, thanks !

четвер, 29 грудня 2016 р. 15:21:47 UTC+2 користувач Rafał Krzewski написав:

[Rafał Krzewski]

Excellent!

Happy hAkking, and all the best in the New Year!

Rafał

[Ashley Aitken]

Hi Evgeny,

Could you briefly describe what you did (or is it exactly as described in links)?  I am interested in doing this too and like you would (have) like(d) to not use another service and zookeeper etc.

I may have been thinking along the same lines as you as well, i.e. having the seed nodes as static instances on the manager nodes, and then scaling further on the worker nodes.

Thanks,

Ashley.

[Evgeny Shepelyuk]

Hello

Well, we stuck to using ConstructR + Zookeeper :)

неділя, 5 березня 2017 р. 11:53:32 UTC+2 користувач Ashley Aitken написав:

[Dr Ashley Aitken]

So you run ConstructR and Zookeeper on all nodes, just manager nodes?

You are still using Docker Swarm?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/6lPAmCWjII4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Evgeny Shepelyuk]

Hello,

1. Yes we're still using Docker Swarm
2. In fact we've postponed AKKA cluster implementation and now we run just a single instance of service, but our plans is 

- introduce ZooKeeper instance (ZooKeeper cluster)
- since ConstructR is a library not a service, it's embedded in the application itself

- so, as soon as we will start having multiple instances - AKKA cluster should work


неділя, 5 березня 2017 р. 18:27:03 UTC+2 користувач Ashley Aitken написав:

[Dr Ashley Aitken]

Thank you very much Evgeny, that makes it a lot clearer and is very helpful.

We can do the same (or similar).

Cheers,

Ashley.

[Evgeny Shepelyuk]

Another example but for Kubernetes. Without using ConstructR.
https://medium.com/google-cloud/clustering-akka-in-kubernetes-with-statefulset-and-deployment-459c0e05f2ea#.vmm894fk1