Akka Microservice platform

[Evan Chan]

Hey guys,

I would like to run an idea by the fine Akka community - which is to discuss what it would take to turn Akka into a platform for building a network of "microservices": each one independently redeployable and easy to change, yet through a common platform, take advantage of the distributed supervision, messaging and other goodies from Akka.

Here are some characteristics of such a platform:

• Service discovery

• Supporting different kinds of data flow topologies - request response, as well as streaming data; pub-sub, etc.

• Provide common abstractions for efficient data serialization

• Support backpressure and flow control, to rate limit requests

• Support easy scaling of each component, including routing of messages or requests to multiple instances

• Enable easy testing of multiple services  (for example, see Akka’s sbt-multi-jvm plugin)

• A common platform for application metrics

• Distributed message or request tracing, to help with visibility and debugging

• Support polyglot development - it should be possible to develop services in different languages

I think many of these are already provided by Akka, but I wanted to run through each one in more detail:

Service Discovery

Right now every actor talks to another actor location-transparently; however, when looking up an external ActorRef, one does have to know the mechanism, ie is it looking up in cluster, or remote, etc....  is it another actorsystem etc...  (this could have changed in 2.2 and 2.3, but I'm not up to date :-p)      What I'm looking for is a mechanism-independent way of looking up actors, remote or not.  IE, I should just need to do this:

   val downstreamActorRef = System.lookupByName(service = "tradingSystem", actor = "masterTrader", ....)

Under the hood this looks up the actorRef using one of configurable mechanisms:

- Akka Cluster is certainly one way to go, with nodes

- At work we use Zookeeper and Curator.  It would be great to make this platform support multiple discovery types

Data Flow Topology

- Akka  is pretty good at this already, supporting many types of data flow.  The only concern I see is that you have to define the flow via the like of routers and such, which are defined in the code on each node, rather than externally via say a message queue (see ZMQ, NSQ etc).  This can be mitigated through DI and configuration and things like that, of course.

Data Serialization

If we are using native Akka protocol to talk over the wire, this is already really good.  One defines case classes, and Akka transparently serializes them over the network if the actor is remote.  This is one thing about Akka that really appeals to me.

So the question is can we make this work for Play / Akka HTTP transparently as well?

    Related - Polyglot support

How would a Ruby/Python/etc process talk to an Akka network?   My thoughts:

- Easiest way would be to have a way to automagically generate HTTP endpoints that includes case class serialization to/from JSON.  Type classes to handle special data types.

- Did you guys define the Akka binary protocol and keep it stable?    Client libraries could then be written for different langauges, but this doesn't solve the problem of message format -- Java serialization and Chill/Kryo won't work.

Backpressure and Flow Control

Reactive streams looks really promising here.  How it ties into routing, topologies, etc. I'd like to find out more about.    

Also, reactive streams won't work for request/response protocols.

Application Metrics and Tracing

Microservices means it becomes more and more important to figure out what's going on across many services.   Fortunately there's a lot of work in this area; multiple third party libs to provide automatic metrics;  somebody wrote an Akka integration with Twitter's Dapper-like tracing system, and I have written a tracing/graphing system as well.

Hot Reloads

I didn't include this in the list above, because the assumption is that with lots of independent small services, they will be naturally easier to redeploy.  Some will argue that the JVM is heavyweight (actually I think a lightly loaded JVM app is under 50MB, which is very reasonable), and will want Erlang-style hot reloads of individual actors.  This is really tricky area though.

Anyways, I'd love to get the thoughts of the community about this idea of using Akka as a "microservice" platform.  

Thanks!

Evan

[Thomas Lockney]

Hey Evan, I'll bite...

I guess my first question would be to ask for a more clear definition of what you mean by a microservice platform. It feels like this is trying to tackle too many different pieces of the ecosystem when some are already well-solved, while others just might not be appropriate to replace with something necessarily Akka-fied. Don't get me wrong, I love Akka, but a lot of this doesn't make a lot of sense to me. 

For example, why would you try to invent create a whole solution for handling polyglot environments when the microservice approach already handles this rather well using RESTful endpoints and/or queue-based solutions. Akka fits in very well with both of these, so trying to find some magic bullet to support it beyond this seems like a losing proposition.

Can you try to clarify what you are envisioning?

~thomas

[Roland Kuhn]

Hi Evan,

thanks for bringing this up! We have been discussing similar things at Akka HQ, so far without the ability to dedicate resources to such an effort. I agree with Thomas that Actors already cover the basics, I would call the Actor Model the ideal substrate on which we can efficiently and conveniently grow uniservices(*).

[specific replies inline]

(*) I prefer this term because it focuses on the Single Responsibility Principle (see also the introduction of modules) instead of the rather irrelevant notion of “size” .

2 jun 2014 kl. 17:51 skrev Evan Chan <vel...@gmail.com>:

Hey guys,

I would like to run an idea by the fine Akka community - which is to discuss what it would take to turn Akka into a platform for building a network of "microservices": each one independently redeployable and easy to change, yet through a common platform, take advantage of the distributed supervision, messaging and other goodies from Akka.

Here are some characteristics of such a platform:

• Service discovery
• Supporting different kinds of data flow topologies - request response, as well as streaming data; pub-sub, etc.
• Provide common abstractions for efficient data serialization
• Support backpressure and flow control, to rate limit requests
• Support easy scaling of each component, including routing of messages or requests to multiple instances
• Enable easy testing of multiple services  (for example, see Akka’s sbt-multi-jvm plugin)
• A common platform for application metrics
• Distributed message or request tracing, to help with visibility and debugging
• Support polyglot development - it should be possible to develop services in different languages

I think many of these are already provided by Akka, but I wanted to run through each one in more detail:

Service Discovery

Right now every actor talks to another actor location-transparently; however, when looking up an external ActorRef, one does have to know the mechanism, ie is it looking up in cluster, or remote, etc....  is it another actorsystem etc...  (this could have changed in 2.2 and 2.3, but I'm not up to date :-p)      What I'm looking for is a mechanism-independent way of looking up actors, remote or not.  IE, I should just need to do this:

   val downstreamActorRef = System.lookupByName(service = "tradingSystem", actor = "masterTrader", ....)

Under the hood this looks up the actorRef using one of configurable mechanisms:

- Akka Cluster is certainly one way to go, with nodes

- At work we use Zookeeper and Curator.  It would be great to make this platform support multiple discovery types

This is indeed the central missing piece, and it is largely orthogonal to the other concerns. My intuitive starting point for the discovery service is that it is basically a function from name to ActorRef that is accessible throughout the deployed application cluster; I would also start from eventually consistent semantics to see whether that fits the bill since everything else will require specially maintained central (SPOF/SPOB) facilities.

Data Flow Topology

- Akka  is pretty good at this already, supporting many types of data flow.  The only concern I see is that you have to define the flow via the like of routers and such, which are defined in the code on each node, rather than externally via say a message queue (see ZMQ, NSQ etc).  This can be mitigated through DI and configuration and things like that, of course.

I’d propose to build only direct peer-to-peer communication into the fabric and model all other concerns (queues, pub-sub, etc.) as uniservices on top. These implementations will then naturally be part of the whole package, but I would rather not conflate different notions of message-passing under a common umbrella.

Data Serialization

If we are using native Akka protocol to talk over the wire, this is already really good.  One defines case classes, and Akka transparently serializes them over the network if the actor is remote.  This is one thing about Akka that really appeals to me.

So the question is can we make this work for Play / Akka HTTP transparently as well?

    Related - Polyglot support

How would a Ruby/Python/etc process talk to an Akka network?   My thoughts:

- Easiest way would be to have a way to automagically generate HTTP endpoints that includes case class serialization to/from JSON.  Type classes to handle special data types.

- Did you guys define the Akka binary protocol and keep it stable?    Client libraries could then be written for different langauges, but this doesn't solve the problem of message format -- Java serialization and Chill/Kryo won't work.

My current thoughts on this front (also for changing the Akka remote protocol to allow interoperability between different versions) go into the direction of CBOR as a kind-of binary JSON-like format that is friendly to streaming, compact, self-describing and standardized. In the end it will require a structural data format (i.e. not tied to JVM nominal types) to accomplish polyglot deployments. The reason to prefer it over protobuf is that it can be parsed without knowing the schema.

The raw serialization implementation can be used for any number of use-cases, e.g. remoting, streams, persistence, etc.

Backpressure and Flow Control

Reactive streams looks really promising here.  How it ties into routing, topologies, etc. I'd like to find out more about.    

Also, reactive streams won't work for request/response protocols.

I’d contest that: I think it a uniservice should not be restricted to A => B, it could also be Stream[A] => Stream[B] (well, these are not the real types). This would take care of the back pressure for data ingesting services in a better fashion than individual calls would, but we need a solution for those as well as you point out. One of the reasons for exclusively targeting direct peer-to-peer communications is that back pressure otherwise is basically unmaintainable.

One related feature is that we should include common patterns like the CircuitBreaker from the get-go.

Application Metrics and Tracing

Microservices means it becomes more and more important to figure out what's going on across many services.   Fortunately there's a lot of work in this area; multiple third party libs to provide automatic metrics;  somebody wrote an Akka integration with Twitter's Dapper-like tracing system, and I have written a tracing/graphing system as well.

Yes, monitoring components must be added as well, I see them as uniservices themselves just like the message routing facilities.

Hot Reloads

I didn't include this in the list above, because the assumption is that with lots of independent small services, they will be naturally easier to redeploy.  Some will argue that the JVM is heavyweight (actually I think a lightly loaded JVM app is under 50MB, which is very reasonable), and will want Erlang-style hot reloads of individual actors.  This is really tricky area though.

I fully agree, redeployment should be done on a per-JVM granularity, where one JVM possibly hosts multiple (related) uniservices.

One thing that is commonly left out—even when citing Erlang—is resilience: it is of the utmost important to include a sane model for handling service failures, restarting or redeploying them, switching to backup topologies, etc. We have that within Akka and porting this to the way we express uniservice architectures is one of the most appealing aspects of this whole endeavor.

Anyways, I'd love to get the thoughts of the community about this idea of using Akka as a "microservice" platform.  

Much appreciated that you started this, and sorry for taking so long to respond, the last few days were a bit hectic on my end.

Regards,

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Evan Chan]

Hey Thomas,

Sorry for the slow reply, for some reason I'm not getting email updates.

RESTful endpoints:  Yes, it is true these handle the polyglot thing very well, assuming standard HTTP and JSON.    However, purely REST based services has many downsides:

- only request/response is supported well.  Doesn't work well for large data / streaming scenarios.

- I suppose this is when you start looking at an Akka-based solution, or message queue for polyglot

- REST services don't handle backpressure very well....   I mean your requests just time out eventually, you need to handle them explicitly

Akka can handle the streaming case very well, but is not polyglot, and is not as easy to interactively test (eg you can't curl Akka endpoints).   This is what I meant by exposing an HTTP layer to Akka.  I guess you can argue that if polyglot is important, then use ZMQ/NSQ/etc.etc.

So what I'm hoping for, is really a friendlier wrapper around Akka, that could potentially enable stuff like polyglot API.  I suppose you can just throw an HTTP endpoint when you need to, which would work too, but I'm talking about something that generates an endpoint for you based on your case classes, thus making it super easy.

-Evan

[Brian]

It seems to me that service discovery could be well addressed by CRDTs, maybe an OR-Set or MC-Set.  In the spare time that I never seem to have, I had thought about grabbing Patrik's CRDT lib and trying to implement service discovery on it to learn more about CRDTs.  Fingers crossed, one of these days i'll get a chance to do it.

I would agree it's a good fit for eventual consistency, if a false positive (service available) is received and the connection fails, a client would move on to the next.  Quorum-based strong consistency ( ZK / Curator ) feels like overkill, but I may be missing an important use case.

Excellent thread, thanks Evan!  Look forward to the discussion on this.

[Patrik Nordwall]

On Mon, Jun 9, 2014 at 2:58 PM, Brian <sculld...@gmail.com> wrote:

It seems to me that service discovery could be well addressed by CRDTs, maybe an OR-Set or MC-Set.  In the spare time that I never seem to have, I had thought about grabbing Patrik's CRDT lib and trying to implement service discovery on it to learn more about CRDTs.  Fingers crossed, one of these days i'll get a chance to do it.

Sounds good.

/Patrik

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Jonas Bonér]

On Mon, Jun 9, 2014 at 2:58 PM, Brian <sculld...@gmail.com> wrote:

It seems to me that service discovery could be well addressed by CRDTs, maybe an OR-Set or MC-Set.  In the spare time that I never seem to have, I had thought about grabbing Patrik's CRDT lib and trying to implement service discovery on it to learn more about CRDTs.  Fingers crossed, one of these days i'll get a chance to do it.

​I like that idea.

​

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
Jonas Bonér
Phone: +46 733 777 123
Home: jonasboner.com
Twitter: @jboner

[Evan Chan]

Hi Brian,

You're thinking of the case where the nodes are not members of an Akka Cluster, right?  So you can keep track of any service. 

How would you detect that a service is down?   ZK has persistent nodes and monitors it's connections (which has both up and downsides).  If an expiry is done with CRDTs that may be part of a solution. 

-Evan

To be free is not merely to cast off one's chains, but to live in a way that respects & enhances the freedom of others. (#NelsonMandela)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/QCYVY-dJlM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Stefan Podkowinski]

Hi

Let me add some notes based on my experience and further thoughts on this.

Am Montag, 2. Juni 2014 17:51:06 UTC+2 schrieb Evan Chan:

Here are some characteristics of such a platform:

• Service discovery

We've been using Zookeeper for discovering individual Akka based services on the network. ZK does this job very well. We tried Akka cluster when it was still experimental. Later we decided that a strong consistency model would be a better fit for the solution we needed to create. But it largely depends on your requirements. If you need to spin up thousands of JVMs then Akka cluster with its gossip protocol will scale very well. However, what we wanted to do is to create a low latency platform using a cell architecture[0], so you'd not have to deal with that many services. In our case a single ZK instance can handle a hundred services just fine.
One important aspect thats just as important as discovery is error handling. Having a single consistent view on your cluster certainly makes it easier to decide whenever a node needs to be removed. From my experience akka cluster using its gossip protocol can sometimes make it hard to find out exactly why nodes have been removed from the cluster (which is important to know for our customers). But thats just anecdotal evidence from my side. Maybe gossip based service discovery can also work fine, but I don't really see the benefits over zookeeper as long as you don't really need to run a very large cluster.

 

• Supporting different kinds of data flow topologies - request response, as well as streaming data; pub-sub, etc.

Very interesting point. But how far would you go? Theres distributed platforms such as Storm [1] (not exactly a micro services platform, I know) that will provide features such as fault tolerance based its idea of how a flow topology should work. On the other hand you're loosing flexibility whenever you need to define a static topology of your services.
Pub-sub is really nice to work with as a developer. But its not suitable for every use-case and can make implementing data flows hard to follow. It would be great to have a platform that would support multiple models based on what you need. In this regard I always liked zeromq[2] for offering you a great deal of options for that, without getting in your way.

 

• Provide common abstractions for efficient data serialization

Akka is pretty good at handling data serialization. Maybe each service description should also specify what kind of serialization protocol is used by the Akka endpoint. But I'd rather see this handled by Akka remoting instead of a microservices framework. E.g. it would be really nice being able to support multiple serialization protocols for your remote actors. Akka would need to figure out which one has been used by the sender and select the appropriate protocol on the receiver end (if supported).
 

• Support backpressure and flow control, to rate limit requests

Lack of back pressure is by far the hardest problem i've came by when dealing with distributed akka applications. Especially for low latency, high throughput systems. Now I'm very happy to see reactive streams happening! Obviously I'm not the only person feeling the pain with this.

However, what does back pressure mean in context of microservices. If you need to call a service and none is available, how will you handle this on framework level? If there's data waiting to be processed but no service running for accepting this data, what does this mean for your data producers? When people talk about back pressure most of the time the discussion is reduced to mailbox congestion and producer/consumer interaction. But if you have a set of services all talking with each other without a well defined topology model, enabling back pressure in terms of transmission throttling for end to end communication isn't enough.
For example, say I have a billing service which would pull pending orders from a queue. Each order would be send to a payment service and afterwards to a mail confirmation service, which in turn also talks to a recommendation service to retrieve a list of items to suggest for further purchase to include in the order confirmation mail. Now in this case, what happens if the recommendation service is down? From a business perspective, its preferred to just keep sending confirmation mails without any recommendations and keep the billing process going. The developer should always be able to decide what to do in case of any end to end service interaction fails. Automatic back pressure could potentially be more dangerous than useful in those situations. In the microservices platform I'd love to use as a developer, I'd always be able to change the way how I interact with services based on the current cluster state and metrics. Services flows should be able to degenerate in case non-critical interactions won't be possible or certain services would just be slow.

 

• Support easy scaling of each component, including routing of messages or requests to multiple instances

Creating solid routing algorithms in a push topology will be though. One possible option would be to take all remote mailbox sizes into account and calculate the average digestion rate. You'd need to consider which service is able to process the message in an acceptable time frame while routing a message to a remote service. Else some kind of backpressure would need to apply.

I'm not a big fan of auto scaling. Deploying new instances should be left to the team. There's alot of great containers nowadays, such as Mesos, which can help you greatly distributing JVM processes. But I'd say in most cases people would start to use microservices on a fixed number of systems. Using Akka theres already alot of options on how to make your application scale. Adding further scale out options as part of a microservices framework is probably not needed.
 

 

• A common platform for application metrics

Again Zookeeper can be very handy for sharing metrics which have been collected using frameworks like codahale metrics. Standard metrics, such as message rates and jvm stats could be provided out of the box by the platform. Custom metrics could be added by the developer. The major question seems to me how metrics should be shared, either in a strong consistency or eventual consistency model. This would depend alot how these values will be used. If you're going to implement load balancing algorithms on top of you metrics (such as message rates) you better make sure to have a recent, consistent picture on whats going on in your cluster.

 

• Distributed message or request tracing, to help with visibility and debugging

Theres already solutions for that [3]. Its probably also more in the scope of the Akka framework itself instead of a microservices framework.
 

• Support polyglot development - it should be possible to develop services in different languages

Might also be out of scope and should be left to Akka how to integrated other languages.

[0] http://highscalability.com/blog/2012/5/9/cell-architectures.html
[1] http://storm.incubator.apache.org/
[2] http://zeromq.org/
[3] https://github.com/levkhomich/akka-tracing/
 

[Brian]

Hi Evan,

In my head it's similar to Netflix's Eureka project, the "server" component would be the cluster and clients could be either cluster members or systems that would use a ClusterClient under the covers to talk to the cluster.   This would then support any service hooked in to the cluster.

I haven't thought a whole lot about detecting how a service is down, but possibly a heartbeat could work like you say with expiry.  Another option could be some kind of health-checky callback from server->client.

I think this approach (vs. a strongly-consistent one) starts to show more promise the more ephemeral services become.  If you think of this uniservice concept (Roland, your royalty check is in the mail) in a dynamically elastic cluster, where many uniservices (check #2 on the way) are going up and down in either a reactive fashion, or a predictive fashion - an eventually consistent approach seems to model the system better.  Spinning a quorum of disks + network traffic would become more and more of an overhead in that use case.

So if we're not focusing on nodes as the unit of service discovery, and instead are focused on the services or more specifically Service Actors running on those nodes - it's not hard for me to imagine needing to track thousands even for a moderately-sized system.

- Brian

[Todd Nist]

Hi,

Here is an alternative approach for service discovery with out using Zookeeper and has better synergy with AKKA clustering; Consul[0].

Consul is based on the Raft protocol[1] for consensus. But Consul also makes uses of Serf’s[2] gossip protocol for LAN and WAN membership status and leader election.

Like etcd and others, Consul's features include service discovery and a similar key-value store, but also provides health checking functionality and the ability to span multiple data-centers out-of-the-box.

The multiple data-center support comes from Consul's use of its eventually consistent gossip protocol. I find this first class support for multiple datacenters to be an interesting leap ahead of other solutions that leave it up to the implementer to figure out.

Perhaps the integration can best be achieved with an akka extension that acts as a client to Consul potentially leveraging either the HTTP API [3] or the roc protocol [4].

With this I believe you could address the different consistency levels expressed (CP + AP) by all participants in this thread so far.  In addition, it can scale to support the situations Brian has described.  If nothing else I think it at least provides some good ideas for service discovery.

Regards,
Todd

[0]  http://www.consul.io/
[1] https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf
[2] http://www.serfdom.io/
[3] http://www.consul.io/docs/agent/http.html
[4] http://www.consul.io/docs/agent/rpc.html

[Evan Chan]

Hi Roland,

Thanks for thoughts...

On Thursday, June 5, 2014 6:57:38 AM UTC-7, rkuhn wrote:

Data Flow Topology

- Akka  is pretty good at this already, supporting many types of data flow.  The only concern I see is that you have to define the flow via the like of routers and such, which are defined in the code on each node, rather than externally via say a message queue (see ZMQ, NSQ etc).  This can be mitigated through DI and configuration and things like that, of course.

I’d propose to build only direct peer-to-peer communication into the fabric and model all other concerns (queues, pub-sub, etc.) as uniservices on top. These implementations will then naturally be part of the whole package, but I would rather not conflate different notions of message-passing under a common umbrella.

Sounds good.  And Akka already has support for ZMQ... 

Data Serialization

If we are using native Akka protocol to talk over the wire, this is already really good.  One defines case classes, and Akka transparently serializes them over the network if the actor is remote.  This is one thing about Akka that really appeals to me.

So the question is can we make this work for Play / Akka HTTP transparently as well?

    Related - Polyglot support

How would a Ruby/Python/etc process talk to an Akka network?   My thoughts:

- Easiest way would be to have a way to automagically generate HTTP endpoints that includes case class serialization to/from JSON.  Type classes to handle special data types.

- Did you guys define the Akka binary protocol and keep it stable?    Client libraries could then be written for different langauges, but this doesn't solve the problem of message format -- Java serialization and Chill/Kryo won't work.

My current thoughts on this front (also for changing the Akka remote protocol to allow interoperability between different versions) go into the direction of CBOR as a kind-of binary JSON-like format that is friendly to streaming, compact, self-describing and standardized. In the end it will require a structural data format (i.e. not tied to JVM nominal types) to accomplish polyglot deployments. The reason to prefer it over protobuf is that it can be parsed without knowing the schema.

The raw serialization implementation can be used for any number of use-cases, e.g. remoting, streams, persistence, etc.

This shouldn't be hard - after all serialization is already pluggable.  Is there momentum behind CBOR though? 

Hot Reloads

I didn't include this in the list above, because the assumption is that with lots of independent small services, they will be naturally easier to redeploy.  Some will argue that the JVM is heavyweight (actually I think a lightly loaded JVM app is under 50MB, which is very reasonable), and will want Erlang-style hot reloads of individual actors.  This is really tricky area though.

I fully agree, redeployment should be done on a per-JVM granularity, where one JVM possibly hosts multiple (related) uniservices.

One thing that is commonly left out—even when citing Erlang—is resilience: it is of the utmost important to include a sane model for handling service failures, restarting or redeploying them, switching to backup topologies, etc. We have that within Akka and porting this to the way we express uniservice architectures is one of the most appealing aspects of this whole endeavor.

You mean Actor become() and friends?    

Failure handling is inherently tied to the topology ...  I'm not sure what common themes we can bring to the table, other than distributed supervision should be a fundamental piece (and a strength of Akka's).

[Evan Chan]

Stefan,

Thanks for your thoughts, replies inlined....

On Monday, June 9, 2014 10:03:38 AM UTC-7, Stefan Podkowinski wrote:

Hi

Let me add some notes based on my experience and further thoughts on this.

Am Montag, 2. Juni 2014 17:51:06 UTC+2 schrieb Evan Chan:

Here are some characteristics of such a platform:

• Service discovery

We've been using Zookeeper for discovering individual Akka based services on the network. ZK does this job very well. We tried Akka cluster when it was still experimental. Later we decided that a strong consistency model would be a better fit for the solution we needed to create. But it largely depends on your requirements. If you need to spin up thousands of JVMs then Akka cluster with its gossip protocol will scale very well. However, what we wanted to do is to create a low latency platform using a cell architecture[0], so you'd not have to deal with that many services. In our case a single ZK instance can handle a hundred services just fine.
One important aspect thats just as important as discovery is error handling. Having a single consistent view on your cluster certainly makes it easier to decide whenever a node needs to be removed. From my experience akka cluster using its gossip protocol can sometimes make it hard to find out exactly why nodes have been removed from the cluster (which is important to know for our customers). But thats just anecdotal evidence from my side. Maybe gossip based service discovery can also work fine, but I don't really see the benefits over zookeeper as long as you don't really need to run a very large cluster.

The key benefit is that with gossip/Akka Cluster, you don't need to maintain ZK at all.  It is true many orgs run ZK anyways, but I've seen enough issues with ZK that I was strongly drawn to a ZK-less solution such as Akka Cluster.   I think the 2.1 preview was just unstable.  Stable gossip systems like Cassandra's is able to give a fairly consistent view....  perhaps some extra protocols are needed on top of Akka Cluster for consistency.... not too sure.

Note that gossip systems may not scale well, depending on the gossip protocol/algorithm.

 

 

• Supporting different kinds of data flow topologies - request response, as well as streaming data; pub-sub, etc.

Very interesting point. But how far would you go? Theres distributed platforms such as Storm [1] (not exactly a micro services platform, I know) that will provide features such as fault tolerance based its idea of how a flow topology should work. On the other hand you're loosing flexibility whenever you need to define a static topology of your services.
Pub-sub is really nice to work with as a developer. But its not suitable for every use-case and can make implementing data flows hard to follow. It would be great to have a platform that would support multiple models based on what you need. In this regard I always liked zeromq[2] for offering you a great deal of options for that, without getting in your way.

ZMQ is nice, except for JNI and you have to install jzmq, a pain.

I guess there is no answer for every situation so a pluggable system like Akka's is fine.

 

 

• Provide common abstractions for efficient data serialization

Akka is pretty good at handling data serialization. Maybe each service description should also specify what kind of serialization protocol is used by the Akka endpoint. But I'd rather see this handled by Akka remoting instead of a microservices framework. E.g. it would be really nice being able to support multiple serialization protocols for your remote actors. Akka would need to figure out which one has been used by the sender and select the appropriate protocol on the receiver end (if supported).

I think Akka URIs now include this no?  Is that what the tcp: thing is for?     Anyways I wasn't suggesting that a micro/uniservice layer have all this functionality, just that somewhere in the stack (could be Akka) it is handled.

 

 

• Support backpressure and flow control, to rate limit requests

Lack of back pressure is by far the hardest problem i've came by when dealing with distributed akka applications. Especially for low latency, high throughput systems. Now I'm very happy to see reactive streams happening! Obviously I'm not the only person feeling the pain with this.

However, what does back pressure mean in context of microservices. If you need to call a service and none is available, how will you handle this on framework level? If there's data waiting to be processed but no service running for accepting this data, what does this mean for your data producers? When people talk about back pressure most of the time the discussion is reduced to mailbox congestion and producer/consumer interaction. But if you have a set of services all talking with each other without a well defined topology model, enabling back pressure in terms of transmission throttling for end to end communication isn't enough.
For example, say I have a billing service which would pull pending orders from a queue. Each order would be send to a payment service and afterwards to a mail confirmation service, which in turn also talks to a recommendation service to retrieve a list of items to suggest for further purchase to include in the order confirmation mail. Now in this case, what happens if the recommendation service is down? From a business perspective, its preferred to just keep sending confirmation mails without any recommendations and keep the billing process going. The developer should always be able to decide what to do in case of any end to end service interaction fails. Automatic back pressure could potentially be more dangerous than useful in those situations. In the microservices platform I'd love to use as a developer, I'd always be able to change the way how I interact with services based on the current cluster state and metrics. Services flows should be able to degenerate in case non-critical interactions won't be possible or certain services would just be slow.

Agreed that developers should be able to pick from different failure strategies.   However it is nice to have some backpressure strategies, like in reactive streams, already implemented.

 

 

• Support easy scaling of each component, including routing of messages or requests to multiple instances

Creating solid routing algorithms in a push topology will be though. One possible option would be to take all remote mailbox sizes into account and calculate the average digestion rate. You'd need to consider which service is able to process the message in an acceptable time frame while routing a message to a remote service. Else some kind of backpressure would need to apply.

Akka cluster measures the JVM metrics of each node, then can publish those numbers through the EventBus, I believe, for use with routers that can variably route messages based on how heavily loaded each client is.   This can easily be part of backpressure. 

[Carsten Saathoff]

Hi,

Am Donnerstag, 5. Juni 2014 15:57:38 UTC+2 schrieb rkuhn:

thanks for bringing this up! We have been discussing similar things at Akka HQ, so far without the ability to dedicate resources to such an effort. I agree with Thomas that Actors already cover the basics, I would call the Actor Model the ideal substrate on which we can efficiently and conveniently grow uniservices(*).

[specific replies inline]

(*) I prefer this term because it focuses on the Single Responsibility Principle (see also the introduction of modules) instead of the rather irrelevant notion of “size” .

I like that term, actually. It describes well what a microservice should be focussed on: doing one thing well.

 

One thing that is commonly left out—even when citing Erlang—is resilience: it is of the utmost important to include a sane model for handling service failures, restarting or redeploying them, switching to backup topologies, etc. We have that within Akka and porting this to the way we express uniservice architectures is one of the most appealing aspects of this whole endeavor.

In addition, I think it should be as transparent as possible. One of the major advantages of RESTful HTTP in microservice architectures is the statelessness of HTTP. A request might be ending up at any endpoint, I don't really have to care, as long as I have a router in between. And routing/loadbalancing HTTP requests is a solved problem. The drawbacks are clearly the increased overhead and that I need to write a layer that translates from messages in one system to JSON, send it via HTTP, receive on the other endpoint and deserialize. 

If I could talk to an actor without knowing where it lives, that would provide the same benefit as the HTTP architecture, but with much less overhead. The cluster sharding allows me to do that, but I would assume that might be a bit too specific as a solution.

And there is another aspect. Especially for larger projects microservices are great to "enforce" a share nothing philosophy. Using akka serialization, I either have to share the messages or use protobuf + code generation, which is again more complicated. I am not sure whether that can be achieved with CBOR standard you mentioned, but using JSON via HTTP has the advantage that I don't have to share code while it's still very easy to use.

But it would be awesome to have an Akka based microservice platform that solves some of the common problems discussed in this thread in a nice way.

best

Carsten

[Vaughn Vernon]

For the service discovery part, it might help to consider the old Jini model. I was going to work on this some time back, but ...

Not that other ideas here aren't valuable, but I think that Jini got a lot of things right. Unfortunately JEE caught the limelight.

Vaughn

[Evan Chan]

Hi Carsten,

On Wednesday, June 11, 2014 2:35:27 AM UTC-7, Carsten Saathoff wrote:

And there is another aspect. Especially for larger projects microservices are great to "enforce" a share nothing philosophy. Using akka serialization, I either have to share the messages or use protobuf + code generation, which is again more complicated. I am not sure whether that can be achieved with CBOR standard you mentioned, but using JSON via HTTP has the advantage that I don't have to share code while it's still very easy to use.

I think the "JSON means you don't have to share code" is a bit false.    Everywhere that uses the JSON interface, especially if it is a compiled language like Java or Scala, is going to need quite a bit of scaffolding, most likely a client, and serialization/deserialization code to work with that REST API.   If you really stick by the "shared nothing", this significant chunk of code must be duplicated, leading to errors and a maintenance nightmare.   It is true that with Akka case classes you _must_ share the code, but I would say in practice this is far easier than having to set up clients and deser code at every client service.

The major advantage of JSON is actually the flexibility of its map-like API means you can be more relaxed in terms of introducing new fields or API changes.   CBOR is like JSON in this respect.

[Carsten Saathoff]

Hi Evan,

Am Freitag, 13. Juni 2014 07:39:15 UTC+2 schrieb Evan Chan:

On Wednesday, June 11, 2014 2:35:27 AM UTC-7, Carsten Saathoff wrote:

And there is another aspect. Especially for larger projects microservices are great to "enforce" a share nothing philosophy. Using akka serialization, I either have to share the messages or use protobuf + code generation, which is again more complicated. I am not sure whether that can be achieved with CBOR standard you mentioned, but using JSON via HTTP has the advantage that I don't have to share code while it's still very easy to use.

I think the "JSON means you don't have to share code" is a bit false.    Everywhere that uses the JSON interface, especially if it is a compiled language like Java or Scala, is going to need quite a bit of scaffolding, most likely a client, and serialization/deserialization code to work with that REST API.   If you really stick by the "shared nothing", this significant chunk of code must be duplicated, leading to errors and a maintenance nightmare.   It is true that with Akka case classes you _must_ share the code, but I would say in practice this is far easier than having to set up clients and deser code at every client service.

I think what you share is the API. Depending on how complicated the API is, it might be absolutely feasible to duplicate the API layer but gain decoupling. With all the good JSON and HTTP libs, writing such a layer is rather easy. I think decoupling of (micro)services is extremely important, and using the same set of case classes means you have an code-level dependency. Using case classes everywhere also means that integrating a service written in another language will be more complicated. But having the freedom to try out new stuff is another big advantage of microservices.

So, don't get me wrong, I am not proposing everything should exclusively communicate via HTTP and JSON, but when we are talking about a microservice platform, I think decoupling of services also on the code level is an extremely important aspect. And I am also aware that HTTP+JSON are not very efficient, so that a more compact alternative such as CBOR sounds like a good idea. And ultimatively, this should probably be configurable, such that in projects where sharing of messages and API code is required, this can be done as well.

 

best

Carsten

[Roland Kuhn]

Hi Evan,

10 jun 2014 kl. 02:26 skrev Evan Chan <vel...@gmail.com>:

Hi Roland,

Thanks for thoughts...

On Thursday, June 5, 2014 6:57:38 AM UTC-7, rkuhn wrote:

Data Flow Topology

- Akka  is pretty good at this already, supporting many types of data flow.  The only concern I see is that you have to define the flow via the like of routers and such, which are defined in the code on each node, rather than externally via say a message queue (see ZMQ, NSQ etc).  This can be mitigated through DI and configuration and things like that, of course.

I’d propose to build only direct peer-to-peer communication into the fabric and model all other concerns (queues, pub-sub, etc.) as uniservices on top. These implementations will then naturally be part of the whole package, but I would rather not conflate different notions of message-passing under a common umbrella.

Sounds good.  And Akka already has support for ZMQ... 

Yes, it should be easy to implement these services.

Data Serialization

If we are using native Akka protocol to talk over the wire, this is already really good.  One defines case classes, and Akka transparently serializes them over the network if the actor is remote.  This is one thing about Akka that really appeals to me.

So the question is can we make this work for Play / Akka HTTP transparently as well?

    Related - Polyglot support

How would a Ruby/Python/etc process talk to an Akka network?   My thoughts:

- Easiest way would be to have a way to automagically generate HTTP endpoints that includes case class serialization to/from JSON.  Type classes to handle special data types.

- Did you guys define the Akka binary protocol and keep it stable?    Client libraries could then be written for different langauges, but this doesn't solve the problem of message format -- Java serialization and Chill/Kryo won't work.

My current thoughts on this front (also for changing the Akka remote protocol to allow interoperability between different versions) go into the direction of CBOR as a kind-of binary JSON-like format that is friendly to streaming, compact, self-describing and standardized. In the end it will require a structural data format (i.e. not tied to JVM nominal types) to accomplish polyglot deployments. The reason to prefer it over protobuf is that it can be parsed without knowing the schema.

The raw serialization implementation can be used for any number of use-cases, e.g. remoting, streams, persistence, etc.

This shouldn't be hard - after all serialization is already pluggable.  Is there momentum behind CBOR though? 

Check Todd Montgomery’s latest: http://www.slideshare.net/toddleemontgomery/qcon-ny-2014-evolving-rest-for-an-iot-world (last slide or so)

I will spend some quality time with RFC7049 in the upcoming weeks, but currently it looks really promising for the reasons I outlined above, and it seems that others have the same impression.

Hot Reloads

I didn't include this in the list above, because the assumption is that with lots of independent small services, they will be naturally easier to redeploy.  Some will argue that the JVM is heavyweight (actually I think a lightly loaded JVM app is under 50MB, which is very reasonable), and will want Erlang-style hot reloads of individual actors.  This is really tricky area though.

I fully agree, redeployment should be done on a per-JVM granularity, where one JVM possibly hosts multiple (related) uniservices.

One thing that is commonly left out—even when citing Erlang—is resilience: it is of the utmost important to include a sane model for handling service failures, restarting or redeploying them, switching to backup topologies, etc. We have that within Akka and porting this to the way we express uniservice architectures is one of the most appealing aspects of this whole endeavor.

You mean Actor become() and friends?    

Failure handling is inherently tied to the topology ...  I'm not sure what common themes we can bring to the table, other than distributed supervision should be a fundamental piece (and a strength of Akka's).

Yes, that is what I meant: we need to lift the notion of supervision from the Actor hierarchy (i.e. within a single uniservice) to the relationship between services. Redeployment etc. is not what become() is about, actors change behavior all the time as part of their normal function. Redeployment would mean crashing / killing the JVM and creating a new one, because only then can you be sure that all resources are cleaned up.

Regards,

Roland

 

Anyways, I'd love to get the thoughts of the community about this idea of using Akka as a "microservice" platform.  

Much appreciated that you started this, and sorry for taking so long to respond, the last few days were a bit hectic on my end.

Regards,

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Roland Kuhn]

This goes in the same direction I am currently exploring: instead of agreeing on the case classes (which means agreeing on fully-qualified class names) we could switch to a more structural approach, meaning that the message itself consists of data that can be parsed without knowing the schema. The recipient can then decide how to respond by querying that structure and extracting what it needs or understands. The gain is that interoperability between different platforms or segregated parts or a larger application or different versions of the same services becomes a lot easier than today. Of course the right content needs to arrive at the right place, but all incidental baggage (like the color of the parcel wrapping—the FQCN) is removed.

RESTful APIs clearly are the inspiration for this line of thinking; are there any fundamental downsides to this approach?

Regards,

Roland

 

best

Carsten

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[√iktor Ҡlang]

IMO using CBOR[1] payloads inside CoAP[2] for device->uniservice comms seems like a really good match.
It would also mesh well (read: be proxyable) with JSON payloads inside REST over HTTP

This would also solve (in a standardized way) service discovery etc. On the backend Akka Cluster paired with a CRDT Map could be used for service registration and would not require any coordination.

1: http://tools.ietf.org/html/rfc7049

2: http://tools.ietf.org/html/draft-ietf-core-coap-18

--

Cheers,

√

[Carsten Saathoff]

Am Samstag, 14. Juni 2014 18:24:34 UTC+2 schrieb rkuhn:

13 jun 2014 kl. 08:27 skrev Carsten Saathoff <car...@kodemaniak.de>: 

Am Freitag, 13. Juni 2014 07:39:15 UTC+2 schrieb Evan Chan:

On Wednesday, June 11, 2014 2:35:27 AM UTC-7, Carsten Saathoff wrote:

And there is another aspect. Especially for larger projects microservices are great to "enforce" a share nothing philosophy. Using akka serialization, I either have to share the messages or use protobuf + code generation, which is again more complicated. I am not sure whether that can be achieved with CBOR standard you mentioned, but using JSON via HTTP has the advantage that I don't have to share code while it's still very easy to use.

I think the "JSON means you don't have to share code" is a bit false.    Everywhere that uses the JSON interface, especially if it is a compiled language like Java or Scala, is going to need quite a bit of scaffolding, most likely a client, and serialization/deserialization code to work with that REST API.   If you really stick by the "shared nothing", this significant chunk of code must be duplicated, leading to errors and a maintenance nightmare.   It is true that with Akka case classes you _must_ share the code, but I would say in practice this is far easier than having to set up clients and deser code at every client service.

I think what you share is the API. Depending on how complicated the API is, it might be absolutely feasible to duplicate the API layer but gain decoupling. With all the good JSON and HTTP libs, writing such a layer is rather easy. I think decoupling of (micro)services is extremely important, and using the same set of case classes means you have an code-level dependency. Using case classes everywhere also means that integrating a service written in another language will be more complicated. But having the freedom to try out new stuff is another big advantage of microservices.

So, don't get me wrong, I am not proposing everything should exclusively communicate via HTTP and JSON, but when we are talking about a microservice platform, I think decoupling of services also on the code level is an extremely important aspect. And I am also aware that HTTP+JSON are not very efficient, so that a more compact alternative such as CBOR sounds like a good idea. And ultimatively, this should probably be configurable, such that in projects where sharing of messages and API code is required, this can be done as well.

This goes in the same direction I am currently exploring: instead of agreeing on the case classes (which means agreeing on fully-qualified class names) we could switch to a more structural approach, meaning that the message itself consists of data that can be parsed without knowing the schema. The recipient can then decide how to respond by querying that structure and extracting what it needs or understands. The gain is that interoperability between different platforms or segregated parts or a larger application or different versions of the same services becomes a lot easier than today. Of course the right content needs to arrive at the right place, but all incidental baggage (like the color of the parcel wrapping—the FQCN) is removed.

RESTful APIs clearly are the inspiration for this line of thinking; are there any fundamental downsides to this approach?

Except what Evan mentioned, I don't think so. I assume it is important to provide a good (de)serialization library. Currently, sending a case class from Actor System A to B is a no-brainer. Probably it will get a bit more complicated when using a more structural approach, since you have to do some manual parsing. So that should be made as simple as possible.

I think the other important question is, how do I address a uniservice, and will stuff like automatic failover, loadbalancing etc. be possible in some standardized way?

best

Carsten

[ch...@ochsnet.com]

On Monday, June 2, 2014 8:51:06 AM UTC-7, Evan Chan wrote:

Hey guys,

I would like to run an idea by the fine Akka community - which is to discuss what it would take to turn Akka into a platform for building a network of "microservices": each one independently redeployable and easy to change, yet through a common platform, take advantage of the distributed supervision, messaging and other goodies from Akka.

    Related - Polyglot support

How would a Ruby/Python/etc process talk to an Akka network?   My thoughts:

- Easiest way would be to have a way to automagically generate HTTP endpoints that includes case class serialization to/from JSON.  Type classes to handle special data types.

- Did you guys define the Akka binary protocol and keep it stable?    Client libraries could then be written for different langauges, but this doesn't solve the problem of message format -- Java serialization and Chill/Kryo won't work.

Thought I'd share how I'm using Jruby with Akka.  My opinion is that the best solution is an idiomatic wrapper around Akka for whatever language you are using.

I created a ruby class that inherits from UntypedActor, and all of my ruby actors inherit from that.  That let me do a few things that were more idiomatic ruby.

Say I have a ruby actor class called MyActor.  To get an actor ref to it, I can do this:

actor_ref = MyActor.find

Or all in one go...

MyActor.find.tell(message) or MyActor.find.ask(message,100)

find takes one argument, the default is the class name.  So if you created the actor with a different name you do this:

MyActor.find("my_other_actor")  (or just Actor::Base.find("my_other_actor")

I can also do MyActor.find_remote(server,name)

The find method actually returns an instance of my own actor ref class.  This was primarily to provide a simple interface to ask/tell, and not have to deal with futures or  ActorSelection/AskableActorSelection directly. 

I also created a simple builder and factory that lets me create actors like so:

Actor::Builder.new(MyActor,arg1,arg2,arg3).with_name("myname").with_router(Javalib::RoundRobinRouter,10).start

It also supports with_parent and with_dispatcher.

The args are called on the actor's post_init method by the factory.

I use protocol buffers for all messages.  I use the protostuf library as I just like it's api better and it's much more idiomatic to use from ruby.

Chris

[Matlik]

Has there been any movement forward on this discussion?  I like the idea of having a single layer of cluster coordination (discovery and availability) for all Akka and non-Akka services.  If there is a clear technical direction and code base, I'd be interested in possibly contributing.  Could/should this be a contrib project, or are the concerns more appropriately addressed in core functionality?

Thanks,

James

[Evan Chan]

The discussion was very long but didn't produce any conclusions.

One possible piece to start with is to make the Akka discovery of other nodes pluggable (whereas the great threshold detection and up/down logic would remain);  for example, maybe the other nodes can be discovered via Consul or ZK instead of purely peer to peer?    This is likely to play better with non-Akka services.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/QCYVY-dJlM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

The fruit of silence is prayer;

the fruit of prayer is faith;

the fruit of faith is love;

the fruit of love is service;

the fruit of service is peace.  -- Mother Teresa

[Ryan Tanner]

We're in the process of moving our Akka cluster over to CoreOS using etcd + skydns for service discovery.  Our dev environment has been running on that for about two weeks now.  Akka nodes publish their network location into etcd and then get their seed nodes out of the same, with skydns mapping logical node names (analytics-1.dev.blah) to their private EC2 IP.  The change comes as part of moving our deployment towards Docker and making it easier to deploy more services along side our cluster (Kafka is up next), all using etcd for service discovery.

For now it's KISS, just using TTLs on the etcd entries to remove downed nodes.  Eventually we might tie that into cluster member events, letting the leader manage the etcd entries (ClusterSingleton probably), we'll see how it goes.

[Evan Chan]

Ryan,

That sounds really interesting.  How are you doing leader election, if I may ask?

Do all nodes register with etcd, or just the seed nodes?   Is there some duplication of state and conflicts, ie Akka detects a node is down but the downed node still registers itself with etcd?

[Ryan Tanner]

At the moment, all nodes register with etcd.  Skydns uses those entries in order to make them addressable by logical hostname.  We're using CoreOS so the actual hostname of the box is just ip-10-0-1-5 or whatever, skydns lets us address them as analytics-1.dev.us-east-1.conspire.local.  IIRC Akka requires nodes to bind to hostname their want to be addressed at, so we need DNS to map those hostnames correctly (unless I misunderstood Akka's networking).  

Right now, the node just publishes its IP and port in etcd and then eventually the skydns instances on every CoreOS box pick up that change.  We aren't doing anything smart to handle downed nodes.  There are TTLs on the etcd entries of 60 seconds and the entries are refreshed every 45 while the node is up.  Fleet ensures that we don't accidentally deploy two nodes of the same name.  If another node picks up a downed node as a potential seed, there might be an issue with that but the node will kill itself if it can't find the cluster. 

I think I'll probably wind up listing seeds elsewhere and controlling that with Akka cluster events while letting nodes still set their own etcd entries for DNS purposes.  There will always be a slight lag in those seed values becoming consistent across the cluster—and I fully recognize that running a cluster on top of a cluster might be slightly nuts—but for the moment I'm content to tackle these issues as they become problems.

Our etcd integration is no way Akka specific right now, we just run a sidekick service using fleet that has a while loop in bash refreshing that etcd entry.  Seed nodes are fetched in the start script which passes them into Akka using -D flags.

[adriano santos]

Hello guys, 

I see that the service discovery may closely resemble with the same approach used by DOSGI. 

http://cxf.apache.org/dosgi-architecture.html

An approach seems to me easier adaptation to AKKA.

Regards,

Adriano Santos

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.