[2.3.2] Akka actor not found through ActorSelection

[Raymond Tay]

Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]

at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]

at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at scala.Option.map(Option.scala:145) [scala-library.jar:na]

Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")

lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")

lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")

lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")

implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 

val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")

 

This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://Trailin...@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

[INFO] [04/28/2014 15:36:58.380] [application-akka.actor.default-dispatcher-3] [akka://application/deadLetters] Message [java.lang.String] from Actor[akka://application/deadLetters] to Actor[akka://application/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[error] play - Cannot invoke the action, eventually got an error: akka.pattern.AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.

[error] application -

! @6i3hf3e84 - Internal server error, for (POST) [/audit] ->

play.api.Application$$anon$1: Execution exception[[AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.]]

at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]

at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at scala.Option.map(Option.scala:145) [scala-library.jar:na]

Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.

at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:132) ~[akka-actor_2.10-2.3.2.jar:na]

at akka.pattern.AskableActorRef$.$qmark$extension(AskSupport.scala:144) ~[akka-actor_2.10-2.3.2.jar:na]

Any ideas guys ? 

Thanks in advance!

Raymond

[Raymond Tay]

Quick update: 

another test i ran this time round is that the cluster application is left running (i.e. Play 2.3 + Akka 2.3.2) and i had a client connect to the clustered application using Play 2.2 + Akka 2.2.3 and this time round i am able to connect to the router and send messages to the environment running Play 2.3 + Akka 2.3.2

Will continue to dig to see what's going on here. 

On Monday, April 28, 2014 3:41:07 PM UTC+8, Raymond Tay wrote:

Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]

at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]

at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at scala.Option.map(Option.scala:145) [scala-library.jar:na]

Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")

lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")

lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")

lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")

implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 

val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")

 

This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://TrailingHammer@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

[Raymond Tay]

Last update:

Workaround is to change the client code on the Play 2.3-M1 side to look like :

object Application extends Controller {

....

val as = ActorSystem("TrailingHammer", ConfigFactory.load("TrailingHammer"))

val backend = as.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")

// use `backend` in callbacks instead

}

Instead of using Play's Akka system. Anyone else met this issue ? Seems like i'm the only one hitting this. Oh well...

[Alon Lavi]

Hey, I'm also having this issue... Any new updates? :)

[Konrad Malawski]

Hi guys,

sorry we didn't reply in this thread (seems we missed it somehow).

In general the code as shown should be working fine in any Akka version... 

Raymond's hint that using a "plain actor system" works may hint at some configuration differences with respect to "play's actor system", hm.

Could one of you perhaps prepare a small app (github?) that would replicate this problem so we can look deeper into config differences etc? That would certainly help locate the problem in here.

On Sat, Oct 11, 2014 at 10:15 AM, Alon Lavi <alon...@gmail.com> wrote:

Hey, I'm also having this issue... Any new updates? :)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Konrad 'ktoso' Malawski

hAkker @ Typesafe

[Ashesh Ambasta]

Same issue with Play 2.3.6 and Akka 2.3.6, spent a day trying all kinds of workarounds and nothing worked.

Proof that technologies like these will take time to become reliable for companies. A day lost looking for hopeless solutions to problems like these is too much for young startups, and this is what makes people like us avoid all of this. Truly disappointed.

On Monday, 28 April 2014 09:41:07 UTC+2, Raymond Tay wrote:

Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]

at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]

at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]

at scala.Option.map(Option.scala:145) [scala-library.jar:na]

Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")

lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")

lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")

lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")

implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 

val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")

 

This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://TrailingHammer@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

[Ryan Tanner]

Well that was a little harsh.

OP (and others): I've definitely run in to what I can only call "weirdness" when trying to rely too heavily on Play's internal actor system for my own purposes.  I've found it's much more reliable to create my own (I use a plugin for this).  

[Roland Kuhn]

Hi Ashesh,

I take that this is the same as the StackOverflow question you opened around the same time. As a quick recap: your issue is that your ActorSystem on the Play side is configured without remoting because otherwise you get bind errors, which is indicative of you creating multiple ActorSystems, possibly inadvertently. In order to get help in these free forums it would be nice to acknowledge that the problem does indeed not lie with the free products you are using.

Otherwise I agree with Ryan in that Play’s default ActorSystem (as exposed by the “Akka” object) is best avoided for the purpose of remoting. Simply create your own ActorSystem using ConfigFactory.loadConfig().getConfig("my-sub-config") and supplying the configuration for your actor system in that config scope.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Patrik Nordwall]

On Wed, Feb 18, 2015 at 1:20 PM, Roland Kuhn <goo...@rkuhn.info> wrote:

Hi Ashesh,

I take that this is the same as the StackOverflow question you opened around the same time. As a quick recap: your issue is that your ActorSystem on the Play side is configured without remoting because otherwise you get bind errors, which is indicative of you creating multiple ActorSystems, possibly inadvertently. In order to get help in these free forums it would be nice to acknowledge that the problem does indeed not lie with the free products you are using.

Otherwise I agree with Ryan in that Play’s default ActorSystem (as exposed by the “Akka” object) is best avoided for the purpose of remoting. Simply create your own ActorSystem using ConfigFactory.loadConfig().getConfig("my-sub-config") and supplying the configuration for your actor system in that config scope.

One thing to be careful with though. When you create your own actor system in play you must also manage its life cycle to avoid resource leaks in development mode, i.e. you must shutdown the actor system when play is reloading.

That is probably the reason why Ryan recommended a plugin.

Regards,

Patrik

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Ryan Tanner]

Indeed.  One thing that is especially tricky is how the lifecycle of your own actor system integrates with the tests.  If you don't clean up properly you can wind up inadvertently reusing actors across test suites which can get quite confusing if they're at all stateful.

[booz....@gmail.com]

Hi Guys

I think this issue is similar to what I am seeing. Could you please shed some light into how I can create my own AkkaSystem on Play instead of Play's.

I am trying to get my Play Controller to communicate with my backend Akka cluster. the html get request is routed to the following method in my Controller:

    public CompletionStage<Result> createSession(int connectionId){
        return FutureConverters.toJava(ask(shardRegion, new CreateSessionMessage(connectionId), 2000))
                .thenApply(response -> ok((String) response));
    }

where

@Inject
    public ConnectionController(ActorSystem system) {
//other things
shardRegion = system.actorSelection("akka.tcp://shar...@127.0.0.1:2551/user/connection"); 
}

My Akka cluster is running and I can see that the three nodes are gossiping with each other constantly.

The problem is I get the following error when createSession is called:play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/deadLetters), Path(/user/connection)]] after [2000 ms]. Message of type [com.vegaspin.actors.connection.ConnectionActorMessage$CreateSessionMessage]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply.]]

My Akka cluster is called "sharding". I am not sure why in the error message it says akka://application/deadLetters

And here is the code for creating the "connection" cluster sharding

    private static ActorRef setupConnectionClusterSharding(ActorSystem actorSystem) {
        ClusterShardingSettings settings = ClusterShardingSettings.create(actorSystem);
        return ClusterSharding.get(actorSystem).start(
                "connection",
                SpringExtProvider.get(actorSystem).props("ConnectionActor"),
                settings,
                ConnectionActorMessage.messageExtractor()
        );
    }