[2.3.2] Akka actor not found through ActorSelection

1534 views
Skip to first unread message

Raymond Tay

unread,
Apr 28, 2014, 3:41:07 AM4/28/14
to akka...@googlegroups.com
Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]
at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]
at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")
lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")
lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")
lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")
implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 
val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")
 
This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://Trailin...@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

[INFO] [04/28/2014 15:36:58.380] [application-akka.actor.default-dispatcher-3] [akka://application/deadLetters] Message [java.lang.String] from Actor[akka://application/deadLetters] to Actor[akka://application/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[error] play - Cannot invoke the action, eventually got an error: akka.pattern.AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.
[error] application -

! @6i3hf3e84 - Internal server error, for (POST) [/audit] ->

play.api.Application$$anon$1: Execution exception[[AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.]]
at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]
at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://application/deadLetters]] had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:132) ~[akka-actor_2.10-2.3.2.jar:na]
at akka.pattern.AskableActorRef$.$qmark$extension(AskSupport.scala:144) ~[akka-actor_2.10-2.3.2.jar:na]


Any ideas guys ? 

Thanks in advance!
Raymond

Raymond Tay

unread,
Apr 28, 2014, 5:14:15 AM4/28/14
to akka...@googlegroups.com
Quick update: 

another test i ran this time round is that the cluster application is left running (i.e. Play 2.3 + Akka 2.3.2) and i had a client connect to the clustered application using Play 2.2 + Akka 2.2.3 and this time round i am able to connect to the router and send messages to the environment running Play 2.3 + Akka 2.3.2

Will continue to dig to see what's going on here. 


On Monday, April 28, 2014 3:41:07 PM UTC+8, Raymond Tay wrote:
Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]
at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]
at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")
lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")
lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")
lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")
implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 
val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")
 
This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://TrailingHammer@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

Raymond Tay

unread,
Apr 28, 2014, 9:37:24 AM4/28/14
to akka...@googlegroups.com
Last update:

Workaround is to change the client code on the Play 2.3-M1 side to look like :
object Application extends Controller {
....
val as = ActorSystem("TrailingHammer", ConfigFactory.load("TrailingHammer"))
val backend = as.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")
// use `backend` in callbacks instead
}

Instead of using Play's Akka system. Anyone else met this issue ? Seems like i'm the only one hitting this. Oh well...

Alon Lavi

unread,
Oct 11, 2014, 4:15:25 AM10/11/14
to akka...@googlegroups.com
Hey, I'm also having this issue... Any new updates? :)

Konrad Malawski

unread,
Oct 12, 2014, 2:55:16 AM10/12/14
to Akka User List
Hi guys,
sorry we didn't reply in this thread (seems we missed it somehow).

In general the code as shown should be working fine in any Akka version... 
Raymond's hint that using a "plain actor system" works may hint at some configuration differences with respect to "play's actor system", hm.
Could one of you perhaps prepare a small app (github?) that would replicate this problem so we can look deeper into config differences etc? That would certainly help locate the problem in here.

On Sat, Oct 11, 2014 at 10:15 AM, Alon Lavi <alon...@gmail.com> wrote:
Hey, I'm also having this issue... Any new updates? :)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Konrad 'ktoso' Malawski
hAkker @ Typesafe


Ashesh Ambasta

unread,
Feb 17, 2015, 8:46:07 AM2/17/15
to akka...@googlegroups.com
Same issue with Play 2.3.6 and Akka 2.3.6, spent a day trying all kinds of workarounds and nothing worked.

Proof that technologies like these will take time to become reliable for companies. A day lost looking for hopeless solutions to problems like these is too much for young startups, and this is what makes people like us avoid all of this. Truly disappointed.


On Monday, 28 April 2014 09:41:07 UTC+2, Raymond Tay wrote:
Hi Akka users,

I've got a situation that's perplexing. It began with Akka 2.3.2 and i have a Play 2.3-M1 attempting to connect to a Akka 2.3.2 actor (part of a cluster named TrailingHammer) and it's not able to do so by throwing an error message like the following:

play.api.Application$$anon$1: Execution exception[[MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)]]
at play.api.Application$class.handleError(Application.scala:297) ~[play_2.10-2.3-M1.jar:2.3-M1]
at play.api.DefaultApplication.handleError(Application.scala:403) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:318) [play_2.10-2.3-M1.jar:2.3-M1]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
Caused by: scala.MatchError: Some(Failure(akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://application/deadLetters), Path(/user/ShadowFrontend)])) (of class scala.Some)

The code in the Play controller looks like this:

lazy val config = ConfigFactory.load("TrailingHammer")
lazy val notificationHost = config.getString("bamboo.persistence.cluster.hostname")
lazy val notificationPort = config.getInt("bamboo.persistence.cluster.port")
lazy val notificationServiceName = config.getString("bamboo.persistence.cluster.name")
implicit lazy val timeOut = Timeout(config.getInt("bamboo.persistence.cluster.query.timeout") seconds) 
val backend = Akka.system.actorSelection(RootActorPath(backendAddress) / "user" / "ShadowFrontend")
 
This code is pretty idiomatic and i've used this successfully in Akka [2.2.3] and wasn't too sure what's going on but the message on Path(/user/ShadowFrontend) seems not to be catching the entire path which should read like "akka.tcp://TrailingHammer@127.0.0.1:2553/user/ShadowFrontend" and when i plugged this string directly into the code and restarted all my services, it gave me the following error message repeatedly indicating that somehow the default actor for handling these faults have died as well. :

Ryan Tanner

unread,
Feb 17, 2015, 12:08:24 PM2/17/15
to akka...@googlegroups.com
Well that was a little harsh.

OP (and others): I've definitely run in to what I can only call "weirdness" when trying to rely too heavily on Play's internal actor system for my own purposes.  I've found it's much more reliable to create my own (I use a plugin for this).  

Roland Kuhn

unread,
Feb 18, 2015, 7:20:30 AM2/18/15
to akka-user
Hi Ashesh,

I take that this is the same as the StackOverflow question you opened around the same time. As a quick recap: your issue is that your ActorSystem on the Play side is configured without remoting because otherwise you get bind errors, which is indicative of you creating multiple ActorSystems, possibly inadvertently. In order to get help in these free forums it would be nice to acknowledge that the problem does indeed not lie with the free products you are using.

Otherwise I agree with Ryan in that Play’s default ActorSystem (as exposed by the “Akka” object) is best avoided for the purpose of remoting. Simply create your own ActorSystem using ConfigFactory.loadConfig().getConfig("my-sub-config") and supplying the configuration for your actor system in that config scope.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.



Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn


Patrik Nordwall

unread,
Feb 19, 2015, 5:36:24 AM2/19/15
to akka...@googlegroups.com
On Wed, Feb 18, 2015 at 1:20 PM, Roland Kuhn <goo...@rkuhn.info> wrote:
Hi Ashesh,

I take that this is the same as the StackOverflow question you opened around the same time. As a quick recap: your issue is that your ActorSystem on the Play side is configured without remoting because otherwise you get bind errors, which is indicative of you creating multiple ActorSystems, possibly inadvertently. In order to get help in these free forums it would be nice to acknowledge that the problem does indeed not lie with the free products you are using.

Otherwise I agree with Ryan in that Play’s default ActorSystem (as exposed by the “Akka” object) is best avoided for the purpose of remoting. Simply create your own ActorSystem using ConfigFactory.loadConfig().getConfig("my-sub-config") and supplying the configuration for your actor system in that config scope.

One thing to be careful with though. When you create your own actor system in play you must also manage its life cycle to avoid resource leaks in development mode, i.e. you must shutdown the actor system when play is reloading.
That is probably the reason why Ryan recommended a plugin.

Regards,
Patrik



--

Patrik Nordwall
Typesafe Reactive apps on the JVM
Twitter: @patriknw

 Scala Days

Ryan Tanner

unread,
Feb 19, 2015, 5:45:43 AM2/19/15
to akka...@googlegroups.com
Indeed.  One thing that is especially tricky is how the lifecycle of your own actor system integrates with the tests.  If you don't clean up properly you can wind up inadvertently reusing actors across test suites which can get quite confusing if they're at all stateful.

booz....@gmail.com

unread,
Apr 26, 2019, 2:33:13 AM4/26/19
to Akka User List
Hi Guys
I think this issue is similar to what I am seeing. Could you please shed some light into how I can create my own AkkaSystem on Play instead of Play's.

I am trying to get my Play Controller to communicate with my backend Akka cluster. the html get request is routed to the following method in my Controller:

    public CompletionStage<Result> createSession(int connectionId){
        return FutureConverters.toJava(ask(shardRegion, new CreateSessionMessage(connectionId), 2000))
                .thenApply(response -> ok((String) response));
    }

where

@Inject
    public ConnectionController(ActorSystem system) {
//other things
shardRegion = system.actorSelection("akka.tcp://shar...@127.0.0.1:2551/user/connection"); 
}

My Akka cluster is running and I can see that the three nodes are gossiping with each other constantly.

The problem is I get the following error when createSession is called:play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/deadLetters), Path(/user/connection)]] after [2000 ms]. Message of type [com.vegaspin.actors.connection.ConnectionActorMessage$CreateSessionMessage]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply.]]

My Akka cluster is called "sharding". I am not sure why in the error message it says akka://application/deadLetters

And here is the code for creating the "connection" cluster sharding

    private static ActorRef setupConnectionClusterSharding(ActorSystem actorSystem) {
        ClusterShardingSettings settings = ClusterShardingSettings.create(actorSystem);
        return ClusterSharding.get(actorSystem).start(
                "connection",
                SpringExtProvider.get(actorSystem).props("ConnectionActor"),
                settings,
                ConnectionActorMessage.messageExtractor()
        );
    }
Reply all
Reply to author
Forward
0 new messages