Akka: Remoting through load balancers
171 views
Skip to first unread message
Ashesh Ambasta
Sep 8, 2015, 1:04:27 PM9/8/15
to Akka User List
Hi guys,
We're a young startup and we decided to stick with Akka for our production deployments. But we recently ran into a minefield of problems trying to set up our Akka applications on our new Mesosphere cluster where we use Marathon to deploy and run applications inside Docker containers.
I've really run out of options at this time and this is my remoting configuration:
# Remoting configuration
app { host = ${?APP_HOST} akka { port = 11112 }}akka {
loglevel = DEBUG
jvm-exit-on-fatal-error = false
log-dead-letters = 1
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = ""
port = ${?app.akka.port}
bind-hostname = ${?app.host}
bind-port = ${?app.akka.port}
}
}
}When other remote actors `ask` an actor remotely, this is what I see in my logs:
[DEBUG] [09/08/2015 16:54:22.909] [application-akka.remote.default-remote-dispatcher-8] [Remoting] Associated [akka.tcp://appli...@10.0.2.23:11112] <- [akka.tcp://appli...@10.0.2.23:11120]
[ERROR] [09/08/2015 16:54:22.925] [application-akka.remote.default-remote-dispatcher-7] [akka.tcp://appli...@10.0.2.23:11112/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010.0.2.23%3A11120-1/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://appli...@centralapp-core.marathon.mesos:11112/]] arriving at [akka.tcp://appli...@centralapp-core.marathon.mesos:11112] inbound addresses are [akka.tcp://appli...@10.0.2.23:11112]What would be the best way to handle this?
I've been trying my best to ask this question on Stackoverflow/IRC, but I've had no replies.
I'll appreciate any help.
Ashesh Ambasta
Sep 8, 2015, 1:06:21 PM9/8/15
to Akka User List
How do you configure Akka not to drop messages?
On Tuesday, 8 September 2015 19:04:27 UTC+2, Ashesh Ambasta wrote:
Hi guys,We're a young startup and we decided to stick with Akka for our production deployments. But we recently ran into a minefield of problems trying to set up our Akka applications on our new Mesosphere cluster where we use Marathon to deploy and run applications inside Docker containers.I've really run out of options at this time and this is my remoting configuration:
loglevel = DEBUG
jvm-exit-on-fatal-error = false
log-dead-letters = 1
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = ""
port = ${?app.akka.port}
bind-hostname = ${?app.host}
bind-port = ${?app.akka.port}
}
}
}When other remote actors `ask` an actor remotely, this is what I see in my logs:
[ERROR] [09/08/2015 16:54:22.925] [application-akka.remote.default-remote-dispatcher-7] [akka.tcp://application@10.0.2.23:11112/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fapplication%4010.0.2.23%3A11120-1/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://application@centralapp-core.marathon.mesos:11112/]] arriving at [akka.tcp://application@centralapp-core.marathon.mesos:11112] inbound addresses are [akka.tcp://application@10.0.2.23:11112]
Ashesh Ambasta
Sep 9, 2015, 4:12:44 AM9/9/15
to Akka User List
In other words, is it possible for Akka to bind remoting to multiple addresses instead of just one?
On Tuesday, 8 September 2015 19:04:27 UTC+2, Ashesh Ambasta wrote:
Hi guys,We're a young startup and we decided to stick with Akka for our production deployments. But we recently ran into a minefield of problems trying to set up our Akka applications on our new Mesosphere cluster where we use Marathon to deploy and run applications inside Docker containers.I've really run out of options at this time and this is my remoting configuration:
loglevel = DEBUG
jvm-exit-on-fatal-error = false
log-dead-letters = 1
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = ""
port = ${?app.akka.port}
bind-hostname = ${?app.host}
bind-port = ${?app.akka.port}
}
}
}When other remote actors `ask` an actor remotely, this is what I see in my logs:
Akka Team
Sep 9, 2015, 8:19:37 AM9/9/15
to Akka User List
Hi Ashesh,
You cannot use Akka Remoting with a load balancer. Any message send targets a specific actor on a specific system (this is what an ActorRef is), you cannot just take them and reroute them to unsuspecting systems.--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Ashesh Ambasta
Sep 9, 2015, 8:25:37 AM9/9/15
to akka...@googlegroups.com
Thank you for your time Endre.
Well, in my opinion, that is something that needs to be documented. Because in my answering actor, I was using `sender()` to reply back. In this case, the use of the word `sender` implies that some reference to the actual actor that sent the message is maintained, whereas, it isn’t.
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/Myn-Pug4Wbg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
Endre Varga
Sep 9, 2015, 8:37:11 AM9/9/15
to akka...@googlegroups.com
Hi Ashesh,
On Wed, Sep 9, 2015 at 2:25 PM, Ashesh Ambasta <ashesh....@gmail.com> wrote:
Thank you for your time Endre.Well, in my opinion, that is something that needs to be documented. Because in my answering actor, I was using `sender()` to reply back. In this case, the use of the word `sender` implies that some reference to the actual actor that sent the message is maintained, whereas, it isn’t.
It is maintained, why do you concluded it isn't? Can you demonstrate with a failing test case where sender() does not work?
The problem with a load balancer is that it reroutes TCP connections, so an unsuspecting actor system thinks it connects to a certains system A while it ends up being connected to B. Then it sends out messages that it thinks should go to A, and then B rightly refuses them since those messages are not destined for B, but A.
-Endre
Ashesh Ambasta
Sep 9, 2015, 8:47:12 AM9/9/15
to akka...@googlegroups.com
In my case, this is the situation:
I’m using Marathon to execute Docker containers with my Akka applications. Now, Marathon, along with Marathon-DNS assigns each set of instances of a running service a DNS, so something like app-a.marathon.mesos (if marathon is running inside mesos). Now, every request to app-a.marathon.mesos goes through an LB which can go to any one of the running instances of app-a.
Now, app-a wants to ask an actor on app-b and app-b replies to the `sender()`, which should be the actor on `app-a`. Now I’m in a soup because:
- I run app-a and bind it to listen for remoting on either the IP of the host its container is running on, in that case, only the reply part works correctly, because the asking actor is associated correctly.
- I run app-a and ask it to bind to `app-a.marathon.mesos`, in which case, I see the sender on `app-b` as `appli...@app-a.marathon.mesos` which is correct but it doesn’t have a specific reference to where the message came from.
I think the assumption is that the binding address per actor system is unique.
Akka Team
Sep 9, 2015, 9:08:50 AM9/9/15
to Akka User List
Hi Ashesh,
These pages describe the addressing scheme:
http://doc.akka.io/docs/akka/2.4.0-RC2/general/addressing.html
http://doc.akka.io/docs/akka/2.4.0-RC2/general/addressing.html
http://doc.akka.io/docs/akka/2.4.0-RC2/general/remoting.html#Peer-to-Peer_vs__Client-Server
(also describe how to separate bind-port from external logical port - this is only available in 2.4-RC2, not in the 2.3 series).
(also describe how to separate bind-port from external logical port - this is only available in 2.4-RC2, not in the 2.3 series).
In general an ActorRef represents a certain actor on a certain machine, not a certain actor on a pool of machines - i.e. it does not work well with external load balancing. There are routers though that solve this at the app level: http://doc.akka.io/docs/akka/2.4.0-RC2/scala/routing.html
If you really want to use external load balancing then you cannot use Remoting with it. In this case you can use any of the other technologies that Akka provides for communication between systems (actor or not):
- akka.streams.io for stream based TCP and UDP APIs (http://doc.akka.io/api/akka-stream-and-http-experimental/1.0/#akka.stream.io.package)
- Spray, for actor based HTTP (http://spray.io)
Remoting is just one of the possible solutions with a certain set of trade-offs. You can even mix-and-match, for example an internal set of systems form a cluster A, using remoting between them, another set of systems form cluster B, using remoting between them, and then A and B communicate via HTTP using load balancing.
-Endre
Ashesh Ambasta
Sep 9, 2015, 9:16:36 AM9/9/15
to akka...@googlegroups.com
Thanks for the input Andre. I actually got quite engrossed with this issue that I ran into since the last two days and what I’m tending towards is that ActorRef based messaging is basically better as a fire-and-forget medium, where you send a message but don’t wait for an immediate reply. Also, ask involves us to have an implicit timeout value at the ask-ing side which may not work well in cases where the remote actor has its own task queuing etc.
I’ll go through all of the possible docs to understand Akka-remoting and its use cases better.
Our workaround now is to deploy single instances of each of the applications that implement ask, and later refactor them to a more reliable communication mechanism.
Moreover, are you aware of any inherent benefits of using ask over any of the other communication mechanisms you’ve mentioned?
Thank you for clarifying this.
Reply all
Reply to author
Forward
0 new messages