Remoting: Service discovery (non-cluster) and specifying hostname/port

[oxbow_lakes]

For our applications which export remote APIs, we tend to follow this mechanism:

• Server discovers free port on localhost
  
• Server binds address into a naming service (JNDI, Zookeeper etc) - under a name - like  "cool-thing/address/Live"
  
• Client uses name to lookup a service. Client can "listen" to name to discover service changes (for example, migration to a new host)
  

The primary driver for this is that no modification should be required for me to migrate a service between hosts 

In Akka, there seem to be a few issues with this:

1. In my application.conf, I have to *explicitly* state what host my server is running on

            netty.tcp {

                hostname = "coolhost.intra.coolco.com"

            }

Why is this? It means that, in order to move my server to startup on a new host I need to either redeploy (because my application.conf is sitting in my source dir) or at the very least edit a config file. I don't even understand the requirement as being something typically required: when I'm running the system, it's implicit that the hostname is *this host*! Using "localhost" doesn't work, however - and neither does ommitting hostname entirely, where you get this error when a client tries to connect:

[ERROR] [12/13/2013 11:23:17.927] [gekkoRemoting-akka.actor.default-dispatcher-7] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A52321-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:20000/]] arriving at [akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:20000] inbound addresses are [akka.tcp://gekkoR...@10.210.51.121:20000]

2. If I let Akka discover a remoting port for me, I can't find out what was chosen (or, if I can, I can't figure out how via the API). This means I cannot bind my location into our naming system - so I have to choose the port myself (not a disaster, of course, but unnecessary)

 

I'm not necessarily talking about clusters here - but simple client/server applications. Apologies if I'm complaining about things which are already possible!

Chris

[Akka Team]

Hi Chris,

On Fri, Dec 13, 2013 at 12:50 PM, oxbow_lakes <oxbow...@gmail.com> wrote:

For our applications which export remote APIs, we tend to follow this mechanism:

• Server discovers free port on localhost
  
• Server binds address into a naming service (JNDI, Zookeeper etc) - under a name - like  "cool-thing/address/Live"
  
• Client uses name to lookup a service. Client can "listen" to name to discover service changes (for example, migration to a new host)
  

The primary driver for this is that no modification should be required for me to migrate a service between hosts 

Sounds entirely reasonable.

 

In Akka, there seem to be a few issues with this:

1. In my application.conf, I have to *explicitly* state what host my server is running on

            netty.tcp {

                hostname = "coolhost.intra.coolco.com"

            }

Why is this?

The actor system’s address must be clearly defined, without any magic in it (because that would make things difficult to understand). Therefore the hostname part must be chosen by the user. This does not mean in any way that you need to edit or even use a config file, you can as well set or override this property when creating the Config which you pass to the ActorSystem upon creation. We do that all the time in our tests:

val config = ConfigFactory.parseString("akka.remote.netty.tcp { hostname=ZAPHOD\n port=42 }").withFallback(ConfigFactor.load())

val system = ActorSystem("Beeblebrox", config)

How you obtain the right hostname string (which can either contain an IP or a resolvable name) depends on the environment you deploy into, therefore Akka cannot help you with that part. Whether a hostname is right is defined to answer the question of whether other hosts can connect to this actor system using that piece of information.

 

It means that, in order to move my server to startup on a new host I need to either redeploy (because my application.conf is sitting in my source dir) or at the very least edit a config file. I don't even understand the requirement as being something typically required: when I'm running the system, it's implicit that the hostname is *this host*! Using "localhost" doesn't work, however - and neither does ommitting hostname entirely, where you get this error when a client tries to connect:

[ERROR] [12/13/2013 11:23:17.927] [gekkoRemoting-akka.actor.default-dispatcher-7] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A52321-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:20000/]] arriving at [akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:20000] inbound addresses are [akka.tcp://gekkoR...@10.210.51.121:20000]

2. If I let Akka discover a remoting port for me, I can't find out what was chosen (or, if I can, I can't figure out how via the API). This means I cannot bind my location into our naming system - so I have to choose the port myself (not a disaster, of course, but unnecessary)

The default address is available on the ActorRefProvider interface, and extensions can access that on the ExtendedActorSystem. Direct drilling looks like

val addr = system.asInstanceOf[ExtendedActorSystem].provider.getDefaultAddress

 

 

I'm not necessarily talking about clusters here - but simple client/server applications. Apologies if I'm complaining about things which are already possible!

Since you normally don’t need to know (and should not care) about transport details—assuming clustering ;-) —the docs contain more info under the topic “how to serialize an ActorRef”.

Regards,

Roland

 

Chris

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[√iktor Ҡlang]

1) You can override configuration with System.properties (command line/startup) so you do not have to put the address or the port in the config file.

[oxbow_lakes]

Hi Roland -

Thanks for this! Unfortunately I can't get the first bit working:

My server-side connect code now looks like this

          val master = ConfigFactory.load()

          val remotingConfig = master.getConfig(s"$mode-remoting")

          def hostName = java.net.InetAddress.getLocalHost.getHostName

          val remoting = ActorSystem("gekkoRemoting", ConfigFactory parseString s"akka.remote.netty.tcp { hostname=$hostName \n port = 0 } " withFallback remotingConfig withFallback master)

          def port = {

            def fromCfg = remotingConfig.getInt("akka.remote.netty.tcp.port")

            remoting.asInstanceOf[ExtendedActorSystem].provider.getDefaultAddress.port getOrElse fromCfg

          }

          val x = remoting.actorOf(Props(new RemotingSupport(port = port)), "repl")

I can see that I get my InetSocketAddress bound correctly (port 57111 chosen by akka)! The server logs this

[INFO] [12/13/2013 13:00:17.610] [system-akka.actor.default-dispatcher-3] [Remoting] Remoting started; listening on addresses :[akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111]

But then I try and connect to this from my client (papagui) - and the server logs this error message

[DEBUG] [12/13/2013 13:01:03.420] [gekkoRemoting-akka.actor.default-dispatcher-6] [Remoting] Associated [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111] <- [akka.tcp://pap...@10.210.51.121:58471]

[ERROR] [12/13/2013 13:01:03.493] [gekkoRemoting-akka.actor.default-dispatcher-14] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A58471-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111/]] arriving at [akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111] inbound addresses are [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111]

Why is my message getting dropped? Is it to do with the fact that ".intra.gsacapital.com" is missing on the server side? If so - why? It's part of the hostname I send

The client connect code is looking up this actor:

      akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111/user/repl

Chris

On Friday, 13 December 2013 12:32:46 UTC, Akka Team wrote:

Hi Chris,

On Fri, Dec 13, 2013 at 12:50 PM, oxbow_lakes <oxbow...@gmail.com> wrote:

For our applications which export remote APIs, we tend to follow this mechanism:

• Server discovers free port on localhost
  
• Server binds address into a naming service (JNDI, Zookeeper etc) - under a name - like  "cool-thing/address/Live"
  
• Client uses name to lookup a service. Client can "listen" to name to discover service changes (for example, migration to a new host)
  

The primary driver for this is that no modification should be required for me to migrate a service between hosts 

Sounds entirely reasonable.

 

In Akka, there seem to be a few issues with this:

1. In my application.conf, I have to *explicitly* state what host my server is running on

            netty.tcp {

                hostname = "coolhost.intra.coolco.com"

            }

Why is this?

The actor system’s address must be clearly defined, without any magic in it (because that would make things difficult to understand). Therefore the hostname part must be chosen by the user. This does not mean in any way that you need to edit or even use a config file, you can as well set or override this property when creating the Config which you pass to the ActorSystem upon creation. We do that all the time in our tests:

val config = ConfigFactory.parseString("akka.remote.netty.tcp { hostname=ZAPHOD\n port=42 }").withFallback(ConfigFactor.load())

val system = ActorSystem("Beeblebrox", config)

How you obtain the right hostname string (which can either contain an IP or a resolvable name) depends on the environment you deploy into, therefore Akka cannot help you with that part. Whether a hostname is right is defined to answer the question of whether other hosts can connect to this actor system using that piece of information.

 

It means that, in order to move my server to startup on a new host I need to either redeploy (because my application.conf is sitting in my source dir) or at the very least edit a config file. I don't even understand the requirement as being something typically required: when I'm running the system, it's implicit that the hostname is *this host*! Using "localhost" doesn't work, however - and neither does ommitting hostname entirely, where you get this error when a client tries to connect:

[ERROR] [12/13/2013 11:23:17.927] [gekkoRemoting-akka.actor.default-dispatcher-7] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A52321-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoRemoting...@LOOWCMARSHA2.intra.gsacapital.com:20000/]] arriving at [akka.tcp://gekkoRemoting@LOOWCMARSHA2.intra.gsacapital.com:20000] inbound addresses are [akka.tcp://gekkoRemoting@10.210.51.121:20000]

2. If I let Akka discover a remoting port for me, I can't find out what was chosen (or, if I can, I can't figure out how via the API). This means I cannot bind my location into our naming system - so I have to choose the port myself (not a disaster, of course, but unnecessary)

The default address is available on the ActorRefProvider interface, and extensions can access that on the ExtendedActorSystem. Direct drilling looks like

val addr = system.asInstanceOf[ExtendedActorSystem].provider.getDefaultAddress

 

 

I'm not necessarily talking about clusters here - but simple client/server applications. Apologies if I'm complaining about things which are already possible!

Since you normally don’t need to know (and should not care) about transport details—assuming clustering ;-) —the docs contain more info under the topic “how to serialize an ActorRef”.

Regards,

Roland

 

Chris

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Endre Varga]

Hi!

[ERROR] [12/13/2013 13:01:03.493] [gekkoRemoting-akka.actor.default-dispatcher-14] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A58471-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111/]] arriving at [akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111] inbound addresses are [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111]

Why is my message getting dropped? Is it to do with the fact that ".intra.gsacapital.com" is missing on the server side? If so - why? It's part of the hostname I send

Yes, the server thinks he has the address [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111] but the inbound message says it is destined to a machine with address [akka.tcp://gekkoR...@LOOWCMARSHA2.intra.gsacapital.com:57111]. Since the two do not match, it thinks it was not sent to him.

-Endre

[oxbow_lakes]

Seems like I need "getCanonicalHostName" when running in test on Windows

On Friday, 13 December 2013 14:46:32 UTC, drewhk wrote:

Hi!

[ERROR] [12/13/2013 13:01:03.493] [gekkoRemoting-akka.actor.default-dispatcher-14] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A58471-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoRemoting...@LOOWCMARSHA2.intra.gsacapital.com:57111/]] arriving at [akka.tcp://gekkoRemoting@LOOWCMARSHA2.intra.gsacapital.com:57111] inbound addresses are [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111]

Why is my message getting dropped? Is it to do with the fact that ".intra.gsacapital.com" is missing on the server side? If so - why? It's part of the hostname I send

Yes, the server thinks he has the address [akka.tcp://gekkoRemoting@LOOWCMARSHA2:57111] but the inbound message says it is destined to a machine with address [akka.tcp://gekkoRemoting@LOOWCMARSHA2.intra.gsacapital.com:57111]. Since the two do not match, it thinks it was not sent to him.

-Endre

 

The client connect code is looking up this actor:

      akka.tcp://gekkoRemoting@LOOWCMARSHA2.intra.gsacapital.com:57111/user/repl

[ahjohannessen]

Roland,

It would probably be a great help if there was some documentation about service discovery and Akka. I often have wondered how people deal with service discovery scenarios in the context of Akka applications. 

I am currently hacking up an akka-etcd client that I intend to use for a similar scenario that Chris lines up. Reason being is that we intend to use Linux containers (docker) and CoreOS as our baseline for all of our services.

On Friday, December 13, 2013 12:32:46 PM UTC, Akka Team wrote:

Hi Chris,

On Fri, Dec 13, 2013 at 12:50 PM, oxbow_lakes <oxbow...@gmail.com> wrote:

For our applications which export remote APIs, we tend to follow this mechanism:

• Server discovers free port on localhost
  
• Server binds address into a naming service (JNDI, Zookeeper etc) - under a name - like  "cool-thing/address/Live"
  
• Client uses name to lookup a service. Client can "listen" to name to discover service changes (for example, migration to a new host)
  

The primary driver for this is that no modification should be required for me to migrate a service between hosts 

Sounds entirely reasonable.

 

In Akka, there seem to be a few issues with this:

1. In my application.conf, I have to *explicitly* state what host my server is running on

            netty.tcp {

                hostname = "coolhost.intra.coolco.com"

            }

Why is this?

The actor system’s address must be clearly defined, without any magic in it (because that would make things difficult to understand). Therefore the hostname part must be chosen by the user. This does not mean in any way that you need to edit or even use a config file, you can as well set or override this property when creating the Config which you pass to the ActorSystem upon creation. We do that all the time in our tests:

val config = ConfigFactory.parseString("akka.remote.netty.tcp { hostname=ZAPHOD\n port=42 }").withFallback(ConfigFactor.load())

val system = ActorSystem("Beeblebrox", config)

How you obtain the right hostname string (which can either contain an IP or a resolvable name) depends on the environment you deploy into, therefore Akka cannot help you with that part. Whether a hostname is right is defined to answer the question of whether other hosts can connect to this actor system using that piece of information.

 

It means that, in order to move my server to startup on a new host I need to either redeploy (because my application.conf is sitting in my source dir) or at the very least edit a config file. I don't even understand the requirement as being something typically required: when I'm running the system, it's implicit that the hostname is *this host*! Using "localhost" doesn't work, however - and neither does ommitting hostname entirely, where you get this error when a client tries to connect:

[ERROR] [12/13/2013 11:23:17.927] [gekkoRemoting-akka.actor.default-dispatcher-7] [akka://gekkoRemoting/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fpapagui%4010.210.51.121%3A52321-0/endpointWriter] dropping message [class akka.actor.SelectChildName] for non-local recipient [Actor[akka.tcp://gekkoRemoting...@LOOWCMARSHA2.intra.gsacapital.com:20000/]] arriving at [akka.tcp://gekkoRemoting@LOOWCMARSHA2.intra.gsacapital.com:20000] inbound addresses are [akka.tcp://gekkoRemoting@10.210.51.121:20000]

2. If I let Akka discover a remoting port for me, I can't find out what was chosen (or, if I can, I can't figure out how via the API). This means I cannot bind my location into our naming system - so I have to choose the port myself (not a disaster, of course, but unnecessary)

The default address is available on the ActorRefProvider interface, and extensions can access that on the ExtendedActorSystem. Direct drilling looks like

val addr = system.asInstanceOf[ExtendedActorSystem].provider.getDefaultAddress

 

 

I'm not necessarily talking about clusters here - but simple client/server applications. Apologies if I'm complaining about things which are already possible!

Since you normally don’t need to know (and should not care) about transport details—assuming clustering ;-) —the docs contain more info under the topic “how to serialize an ActorRef”.

Regards,

Roland

 

Chris

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

[Ryan Tanner]

We use Vagrant and Chef to deploy our Akka cluster on AWS.  At the moment we're using a quite stupid method of service discovery: I wrote a bash script that uses "vagrant ssh" to get the IP of each EC2 instance after it's been created (curl'ing the AWS magic metadata IP) and then update the /etc/hosts on each box to provide it with the EC2 private IPs of the seed nodes.  Any time a box is added or removed, this is re-run.  You might be able to do something similar with docker.

Really, we should use DNS (Route53 + API) or Zookeeper but for some reason this seemed reasonable at two o'clock in the morning.  It's worked out well so far but we have less than ten nodes.

[Chris Marshall]

Well, you can use a combination of CNAME and haproxy in some scenarios but a standardised naming system is very useful. JNDI works but is limited in that 1) it's Java-only, 2) you almost certainly need to write your own implementation (although it's easier than you might think).

For location discovery, you could trivially implement this using JSON over HTTP but ideally you want something more dynamic. But this is an orthogonal problem to akka

Chris

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/kUP6bDTzmNM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Endre Varga]

Hi Ryan,

Is there any reason why you have not used the Amazon EC2 API? At one point we used simply the Java API to discover nodes. 

-Endre

[Ryan Tanner]

That would almost certainly be a better idea, I'm not sure why we didn't think about that.  It came at the end of a week-long push to get our project into production, I'm not sure our brains were fully functioning at that point...