Akka Synchronous Logging

[Nicholas Phillips]

Hello,

I'm trying to establish some logging best practices on my team using Akka.  We're actually in exactly the same boat as the post described here: https://groups.google.com/forum/#!topic/akka-user/ap0bNazxrKo

We're using Java only and SLF4J with logback.  Currently, our actor code is using the LoggingAdapter, and the non-actor code is using straight SLF4J.  This means any actors leveraging the non-actor code are unknowingly paying a synchronous logging penalty in their receive loop.

I'm personally struggling with minor idiosyncrasies between the LoggingAdapter and SLF4J, and at this point, I think I'd like to switch all of my actor code over to straight SLF4J for consistency.  Is this the worst idea ever?  In the post linked above, Patrik mentions

Logging often involve synchronized code and even blocking file IO, i.e. things you should not do in actors without considering the pitfalls

I guess I'd like to obtain a better understanding of what the pitfalls actually are.  Obviously this will negatively impact message throughput. In the case of my system, however, I don't expect to have to deal with thousands (or even hundreds) of messages per second, so I think message throughput is moot for me.  I'm more concerned with stability of the actor system itself - if I get blocked up on file IO, is my actor system going to take a dive, possibly disconnect from the cluster, or something catastrophic?

Alternatively, couldn't I go with an asyncAppender for logback and resolve the problem that way (or pursue log4j2's async logging)?  Any advice would be appreciated.

Thanks,
Nick

[Heiko Seeberger]

Yeah, logging is tricky ;-)

So, you could switch over to using SLF4J or maybe even better Log4j 2 for all of your code. Make sure to use some asynchronous logger or appender (Log4j2 offers very effective asynchronous loggers: https://logging.apache.org/log4j/2.x/manual/async.html).

Then there’s still Akka logging itself via its logging facility. If you’re not interested in Akka’s logging events at all, set the loglevel configuration setting to off. Or maybe better to ERROR.

Me personally I’m still using Akka’s logging facility because of two reasons: First, I’m interested in at least some of Akka’s logging events and I don’t want to deal with two different „channels“. Second, I’m not sure if I really want Log4j/Disruptor to manage some additional threads, just because of the same reasons I don’t want Netty managing threads of its own (and hence I’m looking forward to a Akka IO based rewrite of Akka Remote). Of course this is just a gut feeling and maybe I’m wrong, but OTOH I haven’t run into any logging related issues so far.

Cheers

Heiko

--

Heiko Seeberger

Home: heikoseeberger.de

Twitter: @hseeberger

Public key: keybase.io/hseeberger

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Patrik Nordwall]

Using SLF4J/Logback directly should be fine if you use the async appender. Same with Log4j2, and make sure that you configure it to not block if the ring buffer is full.

/Patrik

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Guido Medina]

You can also use akka-log4j2 in a different way combined with RandomAccessFile (or RollingRandomAccessFile) appender which is synchronous but extremely fast because it has a buffer to minimize I/O (won't block before buffer is full) use with that in mind this is how I got my applications with akka configured:

• If you can always obtain the logger from the actor system as it is an actor hence giving you asynchronous behaviour.
• Configure your logger to be as fast as it can be, don't overkill it with more asynchronous logging which will require an extra thread + LMAX dependency if you do that.
• Make sure you have slf4j-to-log4j2 bridge so that every other API you are using use the same buffer mechanism.

The reason for buffering is because there is a subtle side-effect, akka-logger is an actor handled by the default dispatcher hence when writing to the logger it will block from time to time when the buffer is full which is an acceptable compromise, you couldn't not block if you use asynchronous logger but then you would be losing logs.

I would propose to akka team to add a deployment setting for the logger actor to use a pinned dispatcher and configurable mailbox, you could configure it with a bounded non-blocking mailbox.

Enjoy,

Guido.

[Guido Medina]

Hmmm, though I would be extremely careful not to create a logging loop when the logger actor cannot offer and try to send that message to dead-letters which will also try to log...a configuration that you can say, if a message is sent from this actor don't send it to dead-letters if it fail.

[Nicholas Phillips]

Thanks for the responses, all.

I plan to move forward with the logback AsyncAppender, just because that's what we're already using.  In the event that proves to be problematic, it seems like log4j2's async implementation can squeak out a bit more speed.