How to control excessive logging from LocalActorRefprovider

[GG]

I've got a pretty good handle on configuring Akka logging through a logback.xml file. Almost everything is logging as configured with the exception of LocalActorRefProvider. My system is configured using akka-cluster and it seems that each attempt to look up an actor first goes through LocalActorRefProvider which insists on logging a message about failed path resolution:

[debug] LocalActorRefProvider(akka://ClusterSystem) - resolve of path sequence [/user/token-mgr-proxy/TokenManager#396699396] failed

My logs are littered with these and similar lines. I can't seem to find a way to control the logging from just this one class. I've  muted everything from package akka.actor in my logback.xml and event tried referencing this class directly via its FQCN and via short name (notice it's not printing a FQCN in the log line even though my logback.xml is configured to do so).

Has anyone else run into this problem or have any ideas on how I might be able to resolve the issue?

[GG]

Just to put this in perspective, this logging line generated 6.5GB of logs in about 10 minutes on one of our machines so it's more than just an annoyance for us.

[Roland Kuhn]

Since the log source is specified directly as a String by the LocalActorRefProvider the originating class name is «akka.event.DummyClassForStringSources». The reason is that this LogAdapter needs to be initialized before the ActorSystem is up and running; we could potentially use LogSource.fromClass instead, but that would then not identify the ActorSystem’s address anymore in all cases (especially during start-up where it might be rather interesting). Another possibility would be to add a specific LogSource for this case; if the DummyClass… does not suffice for your purposes, could you file a ticket?

Thanks for raising this,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Patrik Nordwall]

I'm interested in where all those resolve calls come from, because it could be a performance issue. It could be deserialization of ActorRef.

Yesterday I removed storage of the sender ActorRef in persistent events, which typically would result in such things when persistent actors replay the event log.

Does /user/token-mgr-proxy/TokenManager send messages to a persistent actor, or do you know what is causing the failing resolve?

/Patrik

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Brice Figureau]

Hi,

I'm also seeing this log message logged on a very unloaded cluster, on
akka 2.3.11.

On Wed, 2015-06-17 at 13:31 +0200, Patrik Nordwall wrote:
> I'm interested in where all those resolve calls come from, because it
> could be a performance issue. It could be deserialization of ActorRef.

It's possible, but it is hard to confirm, since the log line doesn't
give us a stacktrace to analyze.

It seems to me that it appeared when I started using ClusterSingleton
and ClusterSharding (but I can be wrong). Most of the references of this
log line I see in my logs are for singletons endpoint or sharding
entries.

It looks like it happens approximately at the same time a persistent
actor recovers (which would mean you found the real issue).

> Yesterday I removed storage of the sender ActorRef in persistent
> events, which typically would result in such things when persistent
> actors replay the event log.

Yes, that would make sense.

> Does /user/token-mgr-proxy/TokenManager send messages to a persistent
> actor, or do you know what is causing the failing resolve?

I'll let the OP answer this question.

--
Brice Figureau <bric...@daysofwonder.com>

[Patrik Nordwall]

On Thu, Jun 18, 2015 at 11:46 AM, Brice Figureau <bric...@daysofwonder.com> wrote:

Hi,

I'm also seeing this log message logged on a very unloaded cluster, on
akka 2.3.11.

On Wed, 2015-06-17 at 13:31 +0200, Patrik Nordwall wrote:
> I'm interested in where all those resolve calls come from, because it
> could be a performance issue. It could be deserialization of ActorRef.

It's possible, but it is hard to confirm, since the log line doesn't
give us a stacktrace to analyze.

It seems to me that it appeared when I started using ClusterSingleton
and ClusterSharding (but I can be wrong). Most of the references of this
log line I see in my logs are for singletons endpoint or sharding
entries.

It looks like it happens approximately at the same time a persistent
actor recovers (which would mean you found the real issue).

Yes, that matches my understanding, and that will be solved in 2.4

/Patrik

 

> Yesterday I removed storage of the sender ActorRef in persistent
> events, which typically would result in such things when persistent
> actors replay the event log.

Yes, that would make sense.

> Does /user/token-mgr-proxy/TokenManager send messages to a persistent
> actor, or do you know what is causing the failing resolve?

I'll let the OP answer this question.

--
Brice Figureau <bric...@daysofwonder.com>

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Grant Gavares]

For us, the issue was ultimately a symptom of the problem referenced here: https://groups.google.com/forum/#!topic/akka-user/4lBUX7N7W6k

Our cluster was in a broken state and persistent actors were unable to recover.

Patrik, if I read correctly this issue is resolved in 2.4? If so, would still like me to file a ticket Roland?

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/qFGg0l9-fNg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Patrik Nordwall]

The specific case for persistence replay is fixed, but it would anyway be nice if we used logger names that can filtered. Issue, yes please.
/Patrik

/Patrik