Unobtrusive Logging

[lee.sy...@touchtech.co.nz]

Hi guys,

I wrote a TURN server in Elixir about 18 months ago which works great, but I'm unable to maintain logs in the running application (via Logger) as doing so slows the throughput of the server quite dramatically.  Is there a best practice for unobtrusive logging?  I essentially want to log with as minimal cost to my process flow as possible, as the TURN packets need to be highly optimised in terms of throughput.

Thanks,

Lee

[Robert Virding]

One way to make a much more faster logger is to use the binary rotation logs option in the sasl application. It is much faster than loggers which format their output. Or just log less information. :-)

Robert

[Lee Sylvester]

Thanks, Robert, I'll check there.

I'm not logging much ;-)

Lee

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/VXxb566IUSg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/ff4536b1-ca0a-4903-97f8-fe5572751e8d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Onorio Catenacci]

In my experience, if your logging is actually slowing your app then you're logging a lot. Maybe convert the type of messages so they're not all logged;that is if you have logs at the :debug level consider if some can be :information level instead.

--
Onorio

[Lee Sylvester]

The issue is, there are a lot of packet transformations that occur between receiving a packet and forwarding it on.  I'd like to log as much data as possible in order to be useful to the end user developing their app.  I guess the logging can be switched off when not needed, but it would still be good to capture some data in order to track usage analytics and whatnot.  I guess I was hoping for a simple abstraction solution with minimal overhead.

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/VXxb566IUSg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/6e14e252-aaa3-413c-8966-26e724a21c2e%40googlegroups.com.

[Peter Hamilton]

You could wrap logging calls with a fuse. Cap the number of log entries per minute and throttle if it exceeds that number. Logger already does this to some degree, though you probably want more control than its defaults.

You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAPtDoj16tyNieua2bPzSVNm0q9FJ%2BcBWn6OPizrqya%2BK8WuFbQ%40mail.gmail.com.

[max s]

Hi Lee,

I don't know if you managed to find a solution to the issue you are encountering.

If that's not the case, could you tell us what value do you use in the Logger configuration for the sync_threshold key? Also, what amount of logs does your server generate, and what's the backend you're using?

If you managed to solve it, I think it could be beneficial for the community to know how you did it :) except if you're under some kind of NDA of course.

Max

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAOMhEnyjAVvDzX5YC5H5wJEwqnqW_d%2B54BwLGFzaw0fs8yTOYg%40mail.gmail.com.

[Lee Sylvester]

Hi Max,

Of course, I'm under no NDA.  This is my tech (my company, too), so I know what I can say and what I can't.  ;-)

This is where my ignorance shines through.  I'm not setting a sync_threshold.  I'm using Logger, cus if it's good enough for Jose, it's good enough for me :-P  I've learned to trust that guy.  My config is:

config :logger, 

  compile_time_purge_level: :debug

I'm logging about 4kb of data per second as I'm outputting bytes throughput and some other data.  Nothing huge, and mostly for debug purposes, but the slowdown is evident with about a 25% reduction in throughput along the sockets.  It seems pretty drastic.  The logging is inline, but I doubt that makes a huge amount of difference.

Thanks,

Lee

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAOPKHevV80yTeZuVZj9oKPJRg0hOeZTmbPFTDZzB6gMQLtKLtQ%40mail.gmail.com.

[max s]

Alright you might want to set a value to the sync_threshold.

Please take a look at the source and how events are processed in Elixir Logger: https://github.com/josevalim/logger/blob/master/lib/logger/config.ex#L60-71

When the inbox of the logger is below a certain threshold, it runs asynchronously. Above that threshold, it is synchronous and will slow down the client part (potentially quite heavily depending on the backend you are using).

Try to set the value of that threshold to a thousand ; but be sure to tune that value according to what you are able to observe (how much stress do you need to put on your server such that logging becomes synchronous again for instance, or using the observer).
Anyway in your case, you might need a bigger buffer than the default value ... which is 20 :)

Max

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAPtDoj0orvrEcAYc9B72GKEZs%3Dhxf5HUs8AwHWSY9qcJ297nsA%40mail.gmail.com.