akka debug stream eventually filling up the memory?

[Pierre Andrews]

Hello,

I have been profiling the memory usage of my akka app to get ready to put it in prod, but after a couple hours of running, it eventually runs out of heap. I have currently set the heap to 4Gb, but it runs out of memory with bigger heaps, it's just a question of time.

So, I took a heap dump and opened it with Eclipse Memory Analyser (a bit like jhat, but slightly more user friendly). What I see is a dispatcher taking around around 93% of the used heap (see attached screenshot). If I check what's in that dispatcher's mailbox, it seems to be messages of type akka.event.Logging$Debug (see attached screenshot).

I am now running the app without debug to see if there is some other leak, but I was wondering if there is a way to configure Akka to log the events (so I can see what's going on), but not to retain them forever on the event stream?

Cheers

Pierre

[√iktor Ҡlang]

Hi Pierre,

3 possible solutions:

A) Bulkhead your logger

B) Bound its mailbox so you drop overflow

C) Switch to a more performant logging library (like Logback or otherwise)

Does that help?

Cheers,

√

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cheers,

√

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Pierre Andrews]

Hi Viktor,

this is not coming from my loggers, but from switching on the akka logging:

akka.actor.debug.receive = on
akka.loglevel = "DEBUG"

in the application.conf, so it’s the logging stream of Akka, the doc page: http://doc.akka.io/docs/akka/snapshot/java/logging.html mentions that it’s logged on stdout (and I can change to log4j etc.).

I guess the question is how do I control the mailbox of the akka logging event-stream and how long messages are kept in the stream? A bounded mailbox doesn’t seem to be the right solution as it would drop new log message and keep old messages.

Thanks

Pierre

[√iktor Ҡlang]

Hi Pierre,

Ok. You definitely need to switch to a production-grade logger, you'll find everything you need in the documentation on logging.

Since logging in Akka is just an Actor you can configure the mailbox etc as with any actor.

Happy hAkking!

Cheers,

√

[Josh Hollander]

This even can break with a production grade logger.  I've overflowed the logger's mailbox with Logback.  It seems to me that this behavior breaks the contract of most loggers like Logback by not propagating the actual logging config up to the Akka logger facade.  Simply setting "DEBUG" for all loggers is far less granular than what Logback supports underneath.  If I disable debug in the Logback config for a certain logger then I expect those debug messages should never even be sent to the loggers mailbox.  Even though it's not a good idea to do debug logging in production, it should be possible to set debug level for a certain class of loggers and not completely overwhelm the logger with extraneous messages from other loggers.

-Josh

[Patrik Nordwall]

We have had this ticket regarding more fine grained log level configuration for quite some time. It is not that easy to solve though, because we don't want dependencies to logging backends in akka-actor. 

I would guess that it is not the amount of sent debug events that is the bottleneck, but the actual file writing of the log entries that pass the loglevel filter, and that will cause debug events to be queued in the logger's mailbox.

I just got one idea. Would it help to have a pre-logger actor that only performs the filtering (using backend logger config) before sending the events to the logger actor? Then the discarded log events (debug level) could be garbage collected earlier.

The overhead of constructing the strings of the debug messages on the sender side will still not be solved by that.

Regards,

Patrik

>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html

>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Patrik Nordwall]

I thought a bit harder about this issue, and have come up with a nice solution that will filter log events based on slf4j (logback) config before they are published to the event stream. Pull request here: https://github.com/akka/akka/pull/15413

Cheers,

Patrik

[Ryan Tanner]

Unrelated to the OP's issue, but so long as we're discussing logback, a problem we've always had with logback is it's own debug output during multi-JVM cluster tests.  Logback outputs its status info for every JVM in the test.  Multiple times.  No one on our team has been able to get it to stop, we suspect it's an issue with conflicting logback.xml files on the classpath.

Cmd-f finds the log statement we need, sure, but Circle CI truncates its logs so sometimes the statement is lost. :(

</derail>

[Patrik Nordwall]

Following up here, the solution for fine grained loglevel configuration has been merged to master, and will be included in upcoming (no ETA yet) 2.4-M1.

/Patrik

--

>>>>>>>>>> Read the docs: http://akka.io/docs/

>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html

>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/d/optout.