Akka Spring integration seems to slowly leak memory

[Mark Kaberman]

I have Akka application which is essentially traverses a very large tree where each vertex processing is done by an individual  actor. Different kinds of vertices are processed by different actors. The actors are implemented as prototype Spring beans (they are derived from the same abstract class) and I am using Akka/Spring integration from Akka Spring integration. The only two differences between the githib example and my code is that I use configuration file to configure the routers and the way I get actor references. 

Since the example only uses one actor it creates it like 

system.actorOf(SpringExtProvider.get(system).props("CountingActor"), "counter");

My actor system is hierarchical and I create my actors differently

public ActorRef createActorRef(String actorBeanName, String actorRouterName) {

    ActorRef actor = null;

    final scala.Option<ActorRef> child = context().child(actorRouterName);

    if (child != null && child.isDefined()) {

        actor = child.get();

    } else {

        actor = getContext().actorOf(SpringExtProvider.get(system).props(actorBeanName).withRouter(new FromConfig()), actorRouterName);

    }

}

When my system is running for few days it runs out of memory. I ran the profiler and discovered that there is a huge number of actor's Spring beans being instantiated. So I instrumented my actors with the instance counters:

public abstract class MyActor extends UntypedActor {

    private static AtomicInteger instantiationCount = new AtomicInteger(0);

    public MyActor() {

        logger.info("ACTOR CREATED. Instantiation count {}", instantiationCount.getAndIncrement());

    }

    @Override

    protected void finalize() {

        logger.info("ACTOR FINALIZED. Instantiation count {}", instantiationCount.getAndDecrement());

    }

}

When I run  my application I see constant flow of "ACTOR CREATED" log entries with ever incrementing counter with the finalize() method never called. Eventually after few days of running the system runs out of memory. I am wondering if I am doing something wrong with obtaining the actor references or there is some issues with Akka/Spring integration

[Viktor Klang]

Hi Mark,

Where are you stopping your actors?

--
Cheers,
√

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Mark Kaberman]

Hi Viktor,

I never stop my actors explicitly (except as reaction to failure in supervision strategy). All actors process the vertex data in onReceive() method, determine if a vertex has children, get the children actor by calling my createActorRef, send the message to a child via tell (never ask) and exit onReceive.

Regards,

Mark

[Viktor Klang]

If you create new actors continually and never stop any of them then you have by design got a leak.

--
Cheers,
√

[Mark Kaberman]

Isn't the code:

if (child != null && child.isDefined()) {

       actor = child.get();

}

supposed to fetch existing actor actor reference instead of creating a new one?

If I call stop() at the end of onReceive() what happens to other instances of the same actor which could be processing different vertices? Will they be shut down as well?

[Viktor Klang]

I don't know how your app works or what router config you are using so it is impossible for me to know what's happening.

Perhaps you have Restarts happening which will create new MyActor instances but the old instances are still reachable by something else. Spring perhaps?

Use a memory debugger and trace the reachability of those MyActor instances.

--
Cheers,
√

[Mark Kaberman]

I debugged my application and it seems to be a bug in Akka:

My routers are defined similarly to each other as

/myActor/ {

  dispatcher = my-pinned-dispatcher

  router = round-robin

  nr-of-instances = 10
}

When I debug into my actor creation and messaging method

public sendAkkaMessage(String actorBeanName, String actorRouterName, Object message) {

    ActorRef actor = null;

    final scala.Option<ActorRef> child = context().child(actorRouterName);

    if (child != null && child.isDefined()) {

        actor = child.get();

    } else {

        actor = getContext().actorOf(SpringExtProvider.get(system).props(actorBeanName).withRouter(new FromConfig()), actorRouterName);

    }
    actor.tell(message, self());

}

I see that when a new actor is created by calling getContext().actorOf()...  I see 10 new Spring beans being created (as per Akka config). When an actor is created via child.get() I see one Spring bean is created. So it seems that in the case of a get() Akka disregards already created Spring beans and creates one more. 

[Patrik Nordwall]

I'm 100% sure that the child.get method here doesn't create an actor. How do you see that?

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Mark Kaberman]

I added the static instantiation counter to my actor Spring bean (see my original post). When an actor is created via getContext().ActorOf(... I see number of logging entries coming from the bean's constructor indicating that n instances of the actor beans has been created where n matches nr-of-instances from my Akka config. When context().child.(..).get() is called, I see a single logging entry coming from bean's constructor indicating that a singe actor bean has been created. My logging indicates that  getContext().ActorOf(.. is only called once per actor, but context().child.(..).get()  is called constantly (as it should). So I see ever increasing counter logging indicating that more and more Spring beans are created.

[Viktor Klang]

Is supervision triggering? (i.e. restarts, possibly creating new beans)?

Cheers,

√

[Mark Kaberman]

It does not. 

[Roland Kuhn]

Hi Mark,

it is not easy to help you due to insufficient information:

• How many different actorRouterName arguments can there be? Will the resulting number of actors fit into memory?
• Do you or do you not stop all created actors eventually? Stopping an actor requires a call to context().stop(self()) on every routee.
• You talk about beans, but actors are not beans: where do you create beans and are you certain that you do not retain references that prevent them from being collected? (You should log from postStop() instead of from the finalizer to see whether an actor gets stopped.)
• What is the context in which the sendAkkaMessage method is called? How many of these contexts exist?

Regards,

Roland

[Mark Kaberman]

Hi Roland,

As I wrote in my original post my application traverses the tree. The depth of the three is small and the paths are known in advance, but the breath is very large. So I have relatively small number of actors (and actorRouterName ) arguments

My Akka config looks approximately like:

/rootActor {

 ...

}

/rootActor/myActor1 {

 ...

}

/rootActor/myActor1/myActor2 {

 ...

}

/rootActor/myActor1/myActor2/myActor1 {

 ...

}

- The actors are never stopped. The idea was to reuse the actors. They were only supposed to be stopped as result of supervision strategy triggered by traversal failures.

- My actors are implemented as Spring beans. I am using Akka/Spring integration from here

- sendAkkaMessage called from the context of the parent actor: as part of the vertex discovery I determine if a vertex has children and if yes, of what type. Then I call  sendAkkaMessage  with the proper parameters from the context of an actor which is currently processing the vertex. 

[Roland Kuhn]

14 juni 2016 kl. 16:28 skrev Mark Kaberman <mkab...@gmail.com>:

Hi Roland,

As I wrote in my original post my application traverses the tree. The depth of the three is small and the paths are known in advance, but the breath is very large. So I have relatively small number of actors (and actorRouterName ) arguments

This is only consistent if I assume that the breadth of the tree is small and the depth is very large (because otherwise there would need to be a very large number of actorRouterNames). In this case I guess that you are simply creating way too many actors—and since you never stop them you will eventually drown in them. Using routers of size 10 at every level is going to explode exponentially as you are certainly aware …

In any case: without seeing the full code (which I wouldn’t have time to read) the most sensible hint is that you should use a memory analyzer to verify my guess or find a different explanation (i.e. where all these beans are coming from). Akka for sure does not create anything when calling context().child(...).get() (which is easy to prove since no Props are given).

Regards,

Roland

[Mark Kaberman]

I was able to write a test app which reproduces the problem (attached). Based on the application.conf I would expect 65 instances of the actors to be created. When I start my actors (by executing POST request to http://localhost:8080/start with no body) I see 385 instances created. Which each sequential request extra 385 instances are added until the system runs out of memory. 

[Roland Kuhn]

21 juli 2016 kl. 18:39 skrev Mark Kaberman <mkab...@gmail.com>:

I was able to write a test app which reproduces the problem (attached). Based on the application.conf I would expect 65 instances of the actors to be created. When I start my actors (by executing POST request to http://localhost:8080/start with no body) I see 385 instances created. Which each sequential request extra 385 instances are added until the system runs out of memory. 

I don’t know Spring that well, but could it be that you create one ActorSystem per POST request? (you can verify that by looking at the number of scheduler threads)

Another thing that I’m extremely curious about is why you are making all these actors routers—this just creates a huge number of actors, very likely many more than you have processors in your system. A tree of routers almost never makes sense.

Regards,

Roland

<AkkaDemo.zip>

[Mark Kaberman]

1. Creating an AkkaSystem per POST does not help since each system will still contain large number of actors.

2. I am not sure I understand your comment about the routers. My understanding is that if an actor is defined as 

/someActor {

   router = round-robin-pool

   nr-of-instances = 10

}

means that the round robin router will be used to deliver the messages to 10 instances of someActor and not that someActor is a router

3. I think the problem lies in the way I am defining my actors in the application.conf:  /rootActor/"*"/actor1/"*"/actor5/"*"/actor3 (not the use of the wiled cards. If I define the actor as  /rootActor/actor1/actor5/actor3 the call

actor = getContext().actorOf(SpringExtProvider.get(system).props(actorBeanName).withRouter(new FromConfig()), actorRouterName);

fails with the following exception

[ERROR] [07/22/2016 13:02:14.136] [AkkaDemo-akka.actor.default-dispatcher-3] [akka://AkkaDemo/user/rootActor] configuration problem while creating [akka://AkkaDemo/user/rootActor/$a/actor2] with router dispatcher [akka.actor.default-dispatcher] and mailbox [akka.actor.default-mailbox] and routee dispatcher [akka.actor.default-dispatcher] and mailbox [akka.actor.default-mailbox]

akka.ConfigurationException: configuration problem while creating [akka://AkkaDemo/user/rootActor/$a/actor2] with router dispatcher [akka.actor.default-dispatcher] and mailbox [akka.actor.default-mailbox] and routee dispatcher [akka.actor.default-dispatcher] and mailbox [akka.actor.default-mailbox]

at akka.actor.LocalActorRefProvider.actorOf(ActorRefProvider.scala:797)

at akka.actor.dungeon.Children$class.makeChild(Children.scala:273)

at akka.actor.dungeon.Children$class.actorOf(Children.scala:42)

at akka.actor.ActorCell.actorOf(ActorCell.scala:374)

at org.mark.demo.BaseActor.sendMessage(BaseActor.java:67)

at org.mark.demo.RootActor.onReceive(RootActor.java:38)Enter code here...

[Roland Kuhn]

I'll try to rephrase:

1) I think your problem is that you create one system per call and never shut it down. Don't do that.

2) don't use routers, please.

Sent from my iPhone