Akka not processing messages in mailbox for a very long time after ActorSystem start

[gutzeit]

Dear list members,

Recently we have upgraded to Akka 2.3.14 and I've noticed a very strange behaviour.

Right after startup we start receiving messages on our production platform, and I the logs I see that it takes several minutes for actors start processing messages.

Actors in question are part of routers that are defined in the following way:

/usersRouter {

  router = consistent-hashing-pool

  nr-of-instances = 100000

  virtual-nodes-factor = 10

}

/contactsRouter {

  router = consistent-hashing-pool

  nr-of-instances = 100000

  virtual-nodes-factor = 10

}

Doing jstack in the period that no messages are processed, the only weird thing i can see is the following:

java.lang.Thread.State: RUNNABLE

        at scala.collection.Iterator$class.foreach(Iterator.scala:727)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)

        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)

        at scala.collection.AbstractTraversable.filter(Traversable.scala:105)

        at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:274)

        at scala.collection.AbstractTraversable.filterNot(Traversable.scala:105)

        at akka.routing.RoutedActorCell$$anonfun$1.apply(RoutedActorCell.scala:77)

        at akka.routing.RoutedActorCell$$anonfun$1.apply(RoutedActorCell.scala:77)

        at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)

        at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)

        at scala.collection.Iterator$class.foreach(Iterator.scala:727)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)

        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

        at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)

        at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:105)

        at akka.routing.RoutedActorCell.removeRoutees(RoutedActorCell.scala:77)

        at akka.routing.ResizablePoolCell.resize(Resizer.scala:288)

        at akka.routing.ResizablePoolCell.preSuperStart(Resizer.scala:267)

        at akka.routing.RoutedActorCell.start(RoutedActorCell.scala:123)

        at akka.routing.RoutedActorCell.start(RoutedActorCell.scala:41)

        at akka.actor.RepointableActorRef.point(RepointableActorRef.scala:105)

        at akka.actor.ActorCell.handleSupervise(ActorCell.scala:614)

        at akka.actor.ActorCell.supervise(ActorCell.scala:606)

        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:463)

        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)

        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)

        at akka.dispatch.Mailbox.run(Mailbox.scala:219)

        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)

        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

After that period of "non-processing" is over, that stack trace does not appear anymore.

My best guess is that it is somehow related to latest changes to consistent-hashing pool that was done lately, but then again its just a guess.

Does anyone have any clue what is going on ?

Thanks in advance for your help.

[Akka Team]

Hi,

This might actually be a regression, maybe some initialization code got n^2 unintentionally, and therefore goes wild when used with a large number of actors. I opened a ticket:https://github.com/akka/akka/issues/18613

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[Patrik Nordwall]

From what version did you update?

I can confirm that there is a problem when using pools with many routees (not only consistent hashing pools). May I ask what the use case is for a pool with 100k actors?

Regards,

Patrik

Patrik Nordwall

Typesafe -  Reactive apps on the JVM

Twitter: @patriknw

[Dima Gutzeit]

From 2.1, ancient.

And yes , it's for a pool of 100K.

Regards,
Dima Gutzeit

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/p344JTPpIh8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Roland Kuhn]

Do you have a machine with 100,000 cores? (I’d want one, too!)

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Dima Gutzeit]

It's not for concurrent processing, rather for serialisation of requests on behalf of a single entity. Can't akka handle 100K actors :) ? I remember once reading an article of millions of actors. 

Regards,
Dima Gutzeit

[Justin du coeur]

Akka can handle that many actors easily -- I think the point is that pools per se (as opposed to ordinary actors) are generally used *for* concurrency, as a way of being able to divvy up the work.  What's the reason for having a pool size larger than your possible concurrency factor?

[mathe...@sagaranatech.com]

I, for example, use a large pool in a specific case. I have an actor A that receive a message and emit N requests to another actors, wait all responses, and send them to original requester (a simple aggregate pattern). Instead I create an actor per request, I create an pool with 10k instances and use it to processing my requests. The advantage is not allocate many objects that lead to GC issues. Obviously, the processing in Actor A is very small. 

[gutzeit]

I don't agree with this statement:

"I think the point is that pools per se (as opposed to ordinary actors) are generally used *for* concurrency"

There can be many use cases for Actors in a pool, without concurrency.

Dima.

[Patrik Nordwall]

A workaround until the issue is fixed is to create the actors in two levels.

One pool with 1000 actors, each such is a pool of 100 actors.

/Patrik